Thursday, October 27, 2016

Neural network and deep learning, 3

When you first learn to play golf, you spend most the time developing a basic swing. There are so many available techniques, the best strategy is in-depth study of a few of the most important.

While unpleasant, we also learn quickly when we’re decisively wrong. By contrast, we learn more slowly when our errors are less well-defined.

To address the learning slowdown problem, define cross entropy cost function:

$C = -\frac{1}{n} \sum_x \left[y \ln a + (1-y ) \ln (1-a) \right]$

Thus:

$\frac{\partial C}{\partial w_j} = \frac{1}{n} \sum_x x_j(\sigma(z)-y)$

$\frac{\partial C}{\partial b} = \frac{1}{n} \sum_x (\sigma(z)-y)$

So the rate is proportional to the “error”.

import network2
net = network2.Network([784,30,10],cost=network2.CrossEntropyCost)
net.large_weight_initializer()
net.SGD(training_data,30,10,0.5,evaluation_data=test_data,monitor_evaluation_accuracy=True)

Cross entropy is a measure of surprise.

Softmax

The activation function is changed from sigmoid function to softmax function:

$a^L_j = \frac{e^{z^L_j}}{\sum_k e^{z^L_k}}$

You can think softmax as a way to rescale the weighted input, and form a probability distribution.

The cost function is correspondingly changed to log-likelihood.

overfitting

Fermi was suspicious of models with four free parameters.

We should stop training when accuracy on the validation data is no longer improving.

In general, one of the best ways of reducing overfitting is to increase the size of the training data. However, training data can be expensive or difficult to acquire.

One possible approach is to reduce the size of our network, but it will be less powerful.

We will use a regularization technique called L2/weight decay, which adds a regularization term in the cross-entropy cost:

$C =C_0+\frac{\lambda}{2n} \sum_w w^2= \frac{1}{2n} \sum_x \|y-a^L\|^2 + \frac{\lambda}{2n} \sum_w w^2$

why does regularization help reduce overfitting?

smaller weights -> lower complexity -> simpler explanation for data.

simpler explanation is more resistant to noise.

Such idea as Occam’s Razor is preferred by some people.

However, there’s no a priori logical reason to prefer simple explanation over more complex explanation.

linear fit vs polynomial fit

Newton’s theory vs Einstein’s theory

The true test of a model is not simplicity, but rather how well it does in predicting new phenomena.

So the answer now is that it’s an empirical fact.

Human brain has a huge number of free parameters. Shown just a few images of an elephant, a child will quickly learn to generalize— recognize other elephants. How do we do it? At this point we don’t know.

L1 regularization

$C = C_0 + \frac{\lambda}{n} \sum_w |w|$

The weights shrink by a constant amount.

Dropout

Randomly and temporarily delete half the hidden neurons, restore them after training on a mini-batch of examples. This kind of average or voting scheme is found to be a powerful way of reducing overfitting.

When we dropout different sets of neurons, it’s rather like we’re training different neural networks.This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is therefore forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons.

Dropout is a way of making sure that the model is robust to the loss of any individual piece of evidence.

Artificially expanding the training data

make many small real-world variations, like rotations, translating and skewing the images, elastic distortions,

Neural network vs SVM

With same training data size, Neural network performs better than SVM. However, more traning data can sometimes compensate for differences in the machine learning algorithm used.

Is algorithm A better than algorithm B?

what training data set are you using?

Imagine an alternate world, people created the benchmark data set and have a larger research grant. They might have used the extra money to collect more training data.

The message to take away, is what we want is both better algorithms and better training data.

Weight initialization

Initialize weight of each net as Gaussian random variable with mean 0 and standard deviation 1. This will cause the weighted sum have a standard deviation sqrt(n), leading to activation of the hidden neuron close to either 1 or 0, i.e. saturation.

A better initialization is have individual standard deviation 1/sqrt(n).

import mnist_loader
training_data,validation_data,test_data=mnist_loader.load_data_wrapper()
import network2
net = network2.Network([784,30,10],cost=network2.CrossEntropyCost)
net.SGD(training_data,30,10,0.1,lmbda=5.0,evaluation_data=validation_data, monitor_evaluation_accuracy=True)

How to choose hyper-parameters

You don’t a priori know which hyper-parameters to adjust. If you spend many hours or days or weeks trying this or that, only to get no result, it will damage your confidence.

broad strategy

get any non-trivial learning to achieve results better than chance.
during early stages, make sure you can get quick feedback from experiments.
As with many things in life, getting started can be the hardest thing to do.

variations on SGD

Hessian technique
Momentum-based SGD

other models of artificial neuron

tanh neuron
rectified linear neuron

we don’t hvae a solid theory of how activation functions should be chosen.

Wednesday, October 26, 2016

Neural network and deep learning, 2

2. How the backpropagation algorithm works

Activation $a^l_j $ of the $j^{th}$ neuron in the l layer is related by:

$a^{l}_j = \sigma\left( \sum_k w^{l}_{jk} a^{l-1}_k + b^l_j \right)$

Vectorization will simply it as:

$a^{l} = \sigma(w^l a^{l-1}+b^l)$

The goal of backpropagation is to compute the partial derivatives of the cost function with respect to any weight or bias in the network.

Two assumptions are the cost function is:

an average over individual cost function
a function of the output activations from the neural network.

The error of a neuron is defined as:

$\delta^l_j \equiv \frac{\partial C}{\partial z^l_j}$

where z is the weighted output. We didn’t use $\frac{\partial C}{\partial a^l_j}$ to have a simpler math.

backpropagation based on 4 fundamental equations

error in the output layer(last layer): $\delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j)$ or $\delta^L = (a^L-y) \odot \sigma'(z^L)$
error connected to next layer: $\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l)$
error is the rate of change of the cost with respect to bias: $\frac{\partial C}{\partial b} = \delta$
rate of change of the cost with respect to weight: $\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j$

algorithm:

input x: get activation $a^1$ for input layer
Feedforward: get weighted input and activation for other layers
Output error: compute the error in the last layer by equation 1
Backpropagate the error: using equation 2
Output: get gradient of the cost function by equation 3 and 4.

The backward movement is a consequence of the fact that the cost is a cunction of outputs from the network.

The backpropagatino algorithm is a clever way of keeping track of small perturbation to the weights/biases as they propagate through the network, reach the output, and then affect the cost.

Tuesday, October 25, 2016

Neural network and deep learning, 1

By Michael Nielsen, Jan 2016

The book only has online version due to its interactive graphs, although I would like to have a local copy, either pdf or hard copy.

http://neuralnetworksanddeeplearning.com

official python code

Alternatively, Another more advanced book by MIT is http://www.deeplearningbook.org/ has just published in 2016.12.

[TOC]

What this book is about

Conventional approach to programming: we tell the computer what to do, break big problems up into many small, precisely defined tasks that the computer can easily perform.

Neural network: It learn from observation data, figuring out its own solution to the problem at hand.

Artificial neural networks, which was loosely based on the model of brain neurons, were proposed more 30 years ago. But it was not until the breakthrough in 2006 that the rebranded “deep learning“ become popular. Beside algorithms, many factors collectively contribute to this breakthrough: computing power, large dataset and GPU.

The highlight of this book is that it really focus on a solid understand of the core concepts of neural networks. As he said,

Technologies come and go, but insight is forever.

on the exercises and problems

You should do most of the exercises because they’re basic checks that you’ve understood the material. If you can’t solve an exercise relatively easily, you’ve probably missed something fundamental.

The problems are another matter….With that said, I don’t recommend working through all the problems. What’s even better is to find your own project. Maybe you want to use neural nets to classify your music collection. Or to predict stock prices. Or whatever. But find a project you care about. Then you can ignore the problems in the book, or use them simply as inspiration for work on your own project. Struggling with a project you care about will teach you far more than working through any number of set problems. Emotional commitment is a key to achieving mastery.

1. using neural nets to recognize handwritten digits

2 important types of artificial neuron

perceptron

A perceptron takes several binary inputs, compare the weighted to a threshold value, produces a single binary output. So it’s a device that makes decisions by weighing up evidence:

$\sum_j w_j*x_j>threshold$

For convenience, vector and bias are introduced:

$\overrightarrow{w}\cdot\overrightarrow{x}+b>0$

In biological terms, if the condition is true, it is called fire.

sigmoid neuron

The output is modified by a sigmoid function (or logistic function), defined by

$\sigma(z)=\frac{1}{1+e^{-z}}$

so a large positive z lead to 1, and a large negative z lead to 0

If perceptron is regarded as a step function, then sigmoid neuron is a smoother version.

feedforward neural networks: out from one layer, used as input for next layer

stochastic gradient descent

cost function

$C(w,b)=\frac{1}{2n}\sum_x||y(x)-a||^2$

C is quadratic cost function or the mean squared error.

This cost function is more smooth than counting the correctly recognized numbers, because small changes to the weights and biases won’t change much.

This algorithm which make the cost as small as possible is called gradient descent.

The model is to imagine a ball rolling down to th ebottom of the vally (minimum C), with randomly choosing a starting point. The gradient vector is:

$\Delta C \approx \frac{\partial C}{\partial v_1} \Delta v_1 + \frac{\partial C}{\partial v_2} \Delta v_2\approx \nabla C \cdot \Delta v$

choose a proper learning rate $\eta>0$ and $\Delta v$ , to make a negative gradient:

$\Delta v = -\eta \nabla C$

Then the move of the ball is:

$v \rightarrow v' = v -\eta \nabla C$

The cost function is an average over individual cost, which can take a long time. To speed up learning, stochastic gradient descent is used, which chooses a small sample of randomly chosen training inputs (mini-batch).If only one training input is used, just as human being do, it’s called incremental learning.

code practice

The official MNIST data has 60 k tranning images and 10 k test images. Each training data is 784 dimensions of X and 10 dimensions of y (but the y value of test_data is not one hot encoded). So the number of neurons in the first layer is 784. mini-batch, e.g., each 10 training inputs as a group.

The following codes are run in Python command line. Bascially, it calls Network class from network.py, which defines a sigmoid function for feedforward and the derivative for back propogation.

import mnist_loader
training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
import network
net = network.Network([784, 30, 10])
net.SGD(training_data, 30, 10, 3.0, test_data=test_data)
# (training_data, epochs, mini_batch_size, eta)
#  score ~ 95%

starting point is much more important than algorithms and fine tune in parameters
If making a change improves things, try doing more!
Do you have enough training data?
Do you have enough epochs?
Do you have a good architecture?
Is the learning rate too low or too high?
sophisticated algorithm $$ \leq$$ simple learning algorithm + good training data
they’ve been learned automatically, so we understand neither the brain nor how AI works!
Deep neural networks (5~10 hidden layers) perform far better on many problems than shallow neural network(a single hidden layer).Because deep nets can build up a complex hierarchy of concepts.

Appendix: Is there a simple algorithm for intelligence?

The neural networks can be used to solve pattern recognition problems. The interesting question is will these approaches eventually be used to build thinking machines that match or surpass human intelligence?

Is there a simple set of principles to explain intelligence?

Intelligence may be explained by a large number of fundamentally distant mechanisms, which evolved in response to many different selection pressures in our species’ evolutionary history.

From connectomics, a brain contains 100 billion neurons with 100 trillion connections.If we need to understand the details of all those connections to understand how the brain works, we are not going to have a simple algorithm.

From molecular biology, human and chimp DNA differ at roughly 125 million DNA base pairs, out of a total of 3 billion DNA base pairs. So we are 96% chimp. If we converted these different genetic codes to letters, that’s about 30 times Bible.

Yet our genome alone is not enough to completely describe the neural connections, caveats are growing children need a healthy, stimulating environment and good nutrition to achieve their intelligent potential.

Chimp and human genetic lines diverged just 5 million years ago.

In 1980, Marvin Minsky developed “Society of Mind” theory, proposed that human intelligence is the result of a large society of individually simple computational processes:

What magical trick makes us intelligent? The trick is that there is no trick. The power of intelligence stems from our vast diversity, not from any single, perfect principle.

My own prejudice is in favor of there being a simple algorithm for intelligence. When it comes to research, an unjustified optimism is often more productive than a seemingly better-justified pessimism, for an optimist has the courage to set out and try new things. That’s the path to discovery, even if what is discovered is perhaps not what was originally hoped. A pessimist may be more “correct” in some narrow sense, but will discover less than the optimist.

That’s the path to insight, and by pursuing that path we may one day understand enough to write a longer program or build a more sophisticated network which does exhibit intelligence.

Sunday, October 23, 2016

What a data scientist is like and where they go?

A report by McKinsey Global Institute predicts by 2018 there will be about 1.5 million more jobs than skilled data scientists to fill those jobs. http://jychstar2.blogspot.com/2016/10/careers-in-data-science-nyu.html

Top 5 cities for big data jobs (June, 2014):

San Francisco
Mclean, Va.
Boston
St. Louis
Toronto

Big data professionals can be particularly difficult to find since many roles require a complicated blend of business, analytic, statistical and computer skills — which is not something a candidate acquires overnight. In addition, “clients are looking for people with a certain level of experience, who have worked in a big data environment. There aren’t a lot of them in the market

“What is this person going to be doing? Do you need the technical skills? Or is the quantitative/statistical expertise more important? Is this person going to be doing data modeling or making business decisions?” Kelley says. “In an ideal world, companies want all of it. But it’s not an ideal world.”

First data scientist at Pandora

2001, Gordon Rios was hired by Pandora as the first official data scientist. Three key lessons to take away.

The fully integrated Scientist

He’s completely fascinated with how people determine what to listen to, why and how their tastes and habits change.

He is a full-time member of a Playlist Team, which comes before he is a member of the data science team. He is embeded with a team of engineers, product managers, designers and others. Some companies have all of their DS sitting together, some have them working completely separately from the rest of the company; others follow a consultant-like model where DS parachute into projects temporarily.

Our mission is to make sure that artists to get listeners, and listeners to have the best experience possible.Both depend on getting people to try new music.And that depends on experimenting , collecting data and designing algorithms that push people outside of their musical comfort zones at the right pace.

You need people on operations, engineering, product, and scientists all coming at the problems from different sides, yet with a common vision for the service.

A consultant model would never work. When I first got started with data mining earlier in my career, I often worked as a consultant, and it’s very difficult to make progress on large-scale problems that way. You have to be part of the team to understand all the moving parts.

The best-case scenario is staffing data scientists that have good engineering skills. When DS can ship, you save on headcount and can turn data into meaningful products.

Rios brought ful-stack programming abilities, big data experience, and machine learning expertise to the table. He also had other critical skills that your first scientist needs: the abitliy to work autonomously, self-motivate and be accountable.

In most cases, good management is about lining up people’s skills to the company’s needs, but with data science, so much depends on having people be both skilled and interested.

The art of DS Management

“Of course there are times when you have to be a trooper and tackle a project
that is less than interesting yet critical to the company, but if you have
incredibly rich talent, matching them carefully with the right projects has
got to be what scientific management is all about,” Rios says. “Being able
to consistently do that separates good managers from okay managers.”

Even though they sit separately the majority of the time, the team holds regular
meetings and often rallies for lunch to talk about what they’re working on and
to have more informal conversations about ideas. A lot of solutions emerge outof these discussions. Slightly more formally, they schedule time to present
their projects and findings to their colleagues so they can ask or answer
questions, and/or share practices that might be helpful with other
experiments.

“Success for a data science program is when people are happy, feeling challenged and fulfilled, and delivering important results. That’s when they are at the highest performance state, delivering the most value,” says Rios. “There are a lot of reasons to bring on junior or inexperienced scientists —
they adapt and learn fast — but you better have really solid management and mentorship in place.”

“The hallmark of being a good manager or collaborator is that everyone
wants you involved in their project.” For the perspective employee, they have to bea culture fit firs and thena skill fit. They have to really love the product. and know about the data challenges we’re interested in solving.

Communicate for Optimum Efficiency

“To be an effective data scientist, you have to realize your job isn’t just about the research. You have to quantify and qualify what you do in a way that makes sense across the whole company.

“There are thousands of things we want to try, but we have to work
within the vision for what we want the service to be.”

Building and maintaining a sophisticated A/B testing platform is a must.

Make it standard to foster this transparency every time someone new comes aboard.

This is why we’re so focused on recruiting scientists who are scientifically curious but also entrepreneurial.

Friday, October 21, 2016

Head first C#

metacognition: trick your mind

your brain craves novelty. It’s always searching, scanning, waiting for something unusal.It was built to help you stay alive.
It doesn’t bother saving the boring things; they never make it pass the “this is obviously not important” filter.
How does your brain know what’s improtant. like tiger. like the danger of fire. what happens in your body? Neurons fire. Emotions crank up. Chemical surge. so your brain knows…
Imagine you are siting in a safe, warm, tiger-free zone. You are studying for an exam or trying to learn some tough technical topics which will take a week. Your brian is trying to do you a big favor. The brain’s scarce resources are better spent storing the really big things, not this obviously non-important, non-urgent content.
what turns your brain on? Make it visual. Use a conversational style. think more deeply. touch the emotions. ( you remember what you care about. You remember when you feel something)
the slow way to learn is sheer repetition. The faster way is to do anything that increase your brain activity, especiallly different types of brain activity.

visual application in 10 minutes

I use visual studio 2012, some buttons changes locations in different versions.

new project-> windows forms application, name it. you will get 3 .cs files, of which [design] is used as graphical coding;
toolbox -> pictureBox, drag and drop
rightup corner of the picture has a property arrow -> import image

in Form1.cs, add one line inside the Form1_Load method:

private void Form1_Load(object sender, EventArgs e)
    {
        MessageBox.Show("Contact List 1.0. \nWritten by: Yuchao", "About");
    }

debugging / stop debugging
In Solution Explorer, xx project->add -> New Item-> Local Database, name it
In Database Explorer, xx.sdf, Table ->creat table, set the first column as primary key and true identity, input the desired Database.
Project -> new data source -> database -> dataset -> next -> tables -> Finish
Datasource, drag and drop xx table, customize as you like.
project -> property ->publish -> publish now

.Net Framework

Buttons, checkboxes, lists, etc are all pieces of the .Net framework. It’s got tools to draw grpahiscs, read and write files, manage collections of things. The tools in the .Net framework are divided up into namespaces.

there’s nothing more attactive to a kid than a big sign that says, “Don’t touch this!” Come on, you know you’re tempted…

C# and .Net have lots of buit-in features
IDE chose a namespace for your code, e.g. your project’s namespace
your code is sotred in a class called Program
Every C# program must have exactly one “entry point” method that reads static void Main()
code snippet: for+table+table
when a new object is created from a class, it’s called an instance of that class
static method can be called without an instance
methods in a static class are all static
An object’s hehavior is defined by its methods, and it uses fields to keep track of its state.
Methods are what an object does.
Fields are what the object knows.

Due to time strain, I stop here. But I would definetely go back and spend one week to do the interesting excises in the remaining of the book.