Thursday, December 8, 2016

Machine Learning ND 0, Model Evaluation, Kaggle, machine intelligence 3.0

Udacity also provides a free “Intro to Machine Learning” course, see my previous post
a quick feeling of machine learning algorithm:
  • Decision Tree
  • Naive Bayes
  • Gradient Descent
  • Linear Regression
  • support vector machines
  • neural network
  • k-means clustering
  • hierarchical clustering
  1. stick to your schedule. work regularly.
  2. be relentless in searching for an answer on your own. The struggle and search is where you learn the most. If you come across a term you don’t get, spend time reading up on it.
  3. be active member of your community.

Model Evaluation and Validation

(some course materials are from “introduction to machine learning”)

statistical analysis

Interquartile range (IQR)= Q3-Q1
outlier <Q_1-1.5*IQR
outlier >Q_3 +1.5*IQR

Evaluation Metrics

from sklearn.metrics import mean_absolute_error
from sklearn.metrics import f1_score
from sklearn.metrics import mean_squared_error
In many cases such as disease diagnosis, we don’t care so much about the true negative because it is too common. We care more about the true positive. This can be further divided into postivie predictive value and sensitivity.
e.g. When a search engine) returns 30 pages only 20 of which were relevant while failing to return 40 additional relevant pages, its precision is 20/30 = 2/3 while its recall is 20/60 = 1/3.In this case, Precision is how useful is the result, recall is how complete is the result.
F_\beta =\frac{ (1+\beta^2)  \times precision * recall)} { \beta^2precision + recall}
F_1 =\frac{ 2  \times precision * recall)} { precision + recall}
F1 is criticized because recall and precision are evenly weighted. Therefore, F0.5 focus more on precision, and F2 weights more on recall.

Causes of Error

Bias-variance dilemma:

high bias

a model being unable to represent the complexity of the underlying data
pay little attention to data, oversimplified, high error on trainning set

high variance

a model being overly sensitive to the limited data it has been trained on
pay too much attention to data, overfit, higher error on test set than training set


Ben Hammer:
  • I learn best through creative projects, not lectures.
  • I learn R/Python analytics API’s best through well-crafted examples, not docs
  • Join the competition can be quite addictive. When you make the first submission, you see everyone above you, and then that makes you ask the question what are these guys above me doing? how can I do better than them? And that question keeps striving you to try to do better and better at our problems, and it really forces you to explore the scope of supervised machine learning and different methodologies and approaches that you can use. That means you are not trying one or two ways, you are trying a thousand ways to figure out which make a really big difference in the model performance.
  • 2 catogories for people to approach problems. One spend 2 months trying to develope the great idea, implement it in code and see how it performs. It doesn’t work out.
  • The correct mindset is: I have a lot of different ideas that I think might work out. I want to experiment with them and explore to see how they work. I want to get through as many of these idea as possible, to find the couple really matter and make a big difference. One common pattern amonge winners is that they make 100 to 1000 submissions, they get through athe iterative loop of making a new results. Learning from how it performed very quickly. They optimize their environments and workflows to get through the iterative loop as fast as possible.
The gaming world offers a perfect place to start machine intelligence work (e.g., constrained environments, explicit rewards, easy-to-compare results, looks impressive)—especially for reinforcement learning.
Moral of the story: use a simple model you can understand. Only then move onto something more complex, and only if you need to.
An applied science lab is a big commitment. Data can often be quite threatening to people who prefer to trust their instincts. R&D has a high risk of failure and unusually high levels of perseverance are table stakes. Do some soul searching - will your company really accept this culture?
Data is not a commodity, it needs to be transformed into a product before it’s valuable. Many respondents told me of projects which started without any idea of who their customer is or how they are going to use this “valuable data”. The answer came too late: “nobody” and “they aren’t”

Warm up communication before AI nanodegree

Sebastian Thrun: Ask me anything!
host: Lisbeth Ortega
Ben Kamphaus: “ML and AI algorithms have a role in structuring the way the internet shapes people’s behavior. At present, a lot of this behavior shaping is negative. We see powerful reinforcement for the consumption of false information, distractedness, short-term rewards and less ability to delay gratification. As AI engineers and researchers, what can we do to address this and make technology a better influence on people’s behavior?”monolith education vs. microservices education — decouple the content from the degree and evolve each part individually
I don’t really agree with the premise of this question. Perhaps because I am an optimist. Take Google, for example. Google search machine is a huge powerful AI system. It has given everyone with online access the ability to access an amazing amount of information. While some of this information is clearly false and misleading, I would not want to live in a world without Google.
The use of intentionally misleading fake news has been discussed a lot after the most recent presidential election in the US. But I am an optimist. When we build new technologies, we often don’t get everything right in the beginning. But then we improve. FB for example is now working in leveraging AI to filter out false news items.
“Where do you see ND graduates fitting into the AI industry, which currently seems to be filled with people holding advanced degrees?”
The AI Industry - so to speak - is very large and growing exponentially. Clearly with a Udacity Nanodegree certificate you won’t be as skilled as - say - someone with a PhD from Stanford or MIT. I think of the Nanodegree program much more like a Master’s degree. You get a specific skill set that is highly sought after in Silicon Valley today.
The nice thing about the Nanodegree program is that it is much more up-to-date and relevant than most Master’s programs. This is because the content is directly build with the companies who seek to hire in these areas, like Amazon or IBM Watson.
I firmly believe that AI will be able to make any office worker more efficient. 75% of working Americans work in offices. I would say most of that office work is highly repetitive. Like lawyers when they draft legal documents, or physicians when they diagnose skin diseases.
Examples are Go (AlphaGo), driving cars (Google Self Driving Car), and some of my own Stanford work on finding skin cancer with an iPhone app. (will come out in Nature early next year).
In agriculture, we used to farm fields manually. Hundreds of years ago, nearly all Europeans worked in the farms. One farmer could feed around 4 people or so max. Today a modern farmer can feed hundreds of people, thanks for efficiency gains.
“For someone interested in specializing in computer vision, between AIND and SDCND, which is the right path? In particular, how would you compare the differences in the computer vision aspect of the AIND and SDCND?”
the self driving car Nanodegree is really focused on self driving cars. The computer vision projects all deal with camera data obtained from a car. Very specific, but also highly desired. In AI we take a broader perspective. Computer vision is used in so many other settings, like document scanning, scene analysis, face finding in Snapchat!
There is of course overlap. There is also overlap with out Machine Learning ND, in that machine learning is such a critical component of AI and of Self-Driving Cars.
“I find it hard to believe AI will ever be able to contain creativity or inspiration. Do you think AI will ever develop humour? Do you think this is actually a possibility? Or is AI really just going to be intelligent decision making?”
More broadly: The breakthrough I really want is an AI that sits in my brain and watches me going about my daily activities. And not just me, but perhaps a million other people. And then after watching me for a while, I want AI to simulate me and see if this simulation can fool - say - my family or my coworkers. That would be amazing.
The link between supervised and unsupervised remains vastly underexplored. Many of my Stanford students prime supervised learning tools with massive amounts of unlabeled data. Sometimes they use the algorithms to self-label data, or they pull out different labels only weakly related to the task at hand. In my opinion, most data is kind of unsupervised. But there are still amazing things to be learned from unlabeled data. Look for example at your screen and then move your head around. You are getting training examples on how perspective works. Right there. Wish the field of machine learning spent more time in integrating unlabeled data into supervised learning.
I broadly believe coding will be replaced by training. We will still need people designing architecture, but fewer of us need to be down in the weeds. Instead we will have rich sets of tools. Programming AI will be more like putting those tools together in a meaningful way, and sourcing vast data sets to train these tools.
AI used to be very dogmatic. Researchers in AI would subscribe to one style of AI (eg, non-monotonic logic) and then spend their entire professional career defending it.
This is changing. AI is becoming much more ==pragmatic==. The methods themselves are not as important any more. More important is the architecture and the leverage of data. It’s terrific to see.
Many years ago I revcreated the then-defunct Stanford AI Lab. At that time, Stanford’s AI faculty was divided into subgroups, each of which defended their specific style of AI (eg, knowledge-based systems versus probabilistic networks). This is now a matter of the past. Today the faculty is united and focuses on solving big important societal problem. That’s so great to see.
“In the 20 years since your paper on NeuroChess what leap forward in ML/AI has been the most enlightening to you, for example, Backpropagation, Deep Neural Networks, the link from NNs to Boltzmann - Hamiltonian in physics, Autoencoders or Generative Adversarial Networks?”
First, don’t read my paper on NeuroChess. It’s not my strongest paper :slightly_smiling_face: But more seriously….
wrote by undergrad thesis (Diplomarbeit) on back-propagation and neural networks. I feel the field has come full circle. Neural networks were “out” for a decade. We used to joke at NIPS (Neural Information Processes Conference - the best one out there among the scientific conferences) that the use of “Neural Network” in the title of a submission was highly correlated with a rejection decision. Now it’s hot again. What changed? More data, faster machines. So amazing seeing Yan LeCon’s work receiving so much attention.
I think the original vision - that machines should be trained instead of being programmed - is finally becoming reality. It took 20+ years!!!
If you want to start a strong company: Pick your favorite office job and see if you can leverage AI to make people 10x as efficient. Radiology for example. Totally ripe for disruption.
Impact in small and medium business: Big times IMHO. In the past only very large companies - like Amazon - had the resources to leverage AI. But this is changing now. I remember meeting the founder of Victoria Secret (Les Wexner) and he told me for years he has used machine learning to learn what sells and what doesn’t. He showed me two under garments with a slight different tone in color, and told me one sells at twice the rate of the other. Small and medium business owners often utilize the data they have. But YOU, our Udacity AI students, can change all this!!!!
In general I recommend Russell/Norvig’s AI book. It expensive, but it’s the text every university uses.
Jarek Hirniak: Got it, read it, good, but code is outdated. There was great project on Google Summer of Code to rewrite it in modern language - not sure if it happened.
code for this book:
Agree. This is one of the many reasons why we work with top companies in creating our Nanodegree programs.
I think the biggest revolution will indeed be AI. Most of us do highly repetitive, mindless work all day, for most parts. With the right AI, we can delegate the boring work to the machines, and spend more time being creative.
overlap between Self-Driving car and AI nano degree?
Actually surprisingly little overlap, since the self-driving car ND program is developed by a separate team and focused on the application. The biggest overlap will be machine learning/deep learning and specifically image understanding.
As a potential student of the AI nanodegree, should I read that book before? After? Or it would be just a reference manual at that point?
Don’t read the book before. It’s really just a good reference.
The new version of the book?
Yes he is. I don’t know when it’s coming out. I hear it will be a substantial revision, not just a minor one. With the recent success of machine learning, much of the other content has become less relevant.
It’s mentioned that IBM Watson and Didi Chuxing are your hiring partners. Could you elaborate more on that? How you make hiring decisions?
We told a few “friends” about this program, and pretty much every company we talk to is eager to hire. Hiring partners are willing to let us say so publicly, and use their logo etc. In return they get preferred access to our student CVs (when students opt in). No company guarantees jobs here - but our partners do pay attention to the Nanodegree credential. To be totally honest here: Udacity’s intent here is to create a new pipeline for individuals to find a job in the tech industry. Not everyone is admitted into MIT or CMU. Since we started working with hiring partners and partner companies, hundreds of students have found jobs already. Over 20 now at Google! So it seems to be working …. knock on wood … still a long way to go ….
But to be clear: our hiring partners make the final hiring decisions.
“Ideally in the future, you’ve envisioned nanodegrees supplanting traditional university models by supplying targeted, specialized skillsets oriented for application. How do you envision university models changing in your vision for the future if nanodegrees take off?”
I believe NDs are complimentary to college education. The typical ND student is a lifelong learner (although we also have many college students who want contemporary skills). I believe higher education in general should be with us people for our entire professional lives, not just a short initial phase. Turns out past college degree, Udacity is perhaps the best way to get contemporary education and start a new career - well - I might be a bit biased here….

udacity talk 4, Erikc Darnell

udacity talk 4, Erikc Darnell
You look so amazing and accomplished. You just don’t have to practice anymore?
Actually I practice 5 hours a day. If I don’t, I start to get bad really fast.
passion question.
Everybody is different. For me, I think it just took up a lot of maturation and a lot of stuff just didn’t come naturally to me that I have to learn. How long will it take to get you there? You will never get there if you don’t start going. You can’t win a game until you start. I always encourage people to just go and do it. Go to your studio and make stuff.
Today we have very cheap camera. In the old time, we only had film. That actually worked well for me. Because every time you pull the trigger when you’re shooting motion picture film, it’s like dollar signs.. So you have to really think about what it is you’re shooting.
When camera comes out, you tend to shoot everything all day long. But you never look at it at the second time.
How do you promote in a highly innovative environment?
Get people involve in every level. Great idea can come from anybody. If you have an environment where people feel free to communicate, not only with their peers, but with the CEO, even the person that stock in the refrigerator. He is a part of this team and feels accepted and he is inside. If the person is confident enough to raise your hand and say, what if you did it this way? Then you got an environment that empower innovation because everybody feels like they have permission.
Most challenging thing for directing a movie ? It’s always story. People want something deeper ultimately. It doesn’t matter how cool the tech if you have very few content.
I learned a lot from undergraduate, learn a great deal from calart. It’s great opportunity to really focus yourself. But to be a filmmaker and in particular to be a director, you have to go through thw world with an open mind and open eyes. It’s important to be able to talke the language of the programmer, the language of music composer, voice actor. It’s kind of like “a jack of all trades, a master of none”. You try to surround yourself with people that are better what they do than you could ever be. That’s the secret key. Find someone ambitious than you and become their partner.
Advice for young? The world is always changing. The best way to navigate that it’s not to necessarily think that every single detail of your life to be planned out, or you have to make the right choice every step of the way. The best thing is Joseph Campbell says, just follow your bliss, find the things that are most exciting and interesting to you, because then you’re going to do your best work and you’re going to learn the most, because you care. You develop things from passion.