Tuesday, October 25, 2016

Neural network and deep learning, 1

By Michael Nielsen, Jan 2016

The book only has online version due to its interactive graphs, although I would like to have a local copy, either pdf or hard copy.

http://neuralnetworksanddeeplearning.com

official python code

Alternatively, Another more advanced book by MIT is http://www.deeplearningbook.org/ has just published in 2016.12.

[TOC]

What this book is about

Conventional approach to programming: we tell the computer what to do, break big problems up into many small, precisely defined tasks that the computer can easily perform.

Neural network: It learn from observation data, figuring out its own solution to the problem at hand.

Artificial neural networks, which was loosely based on the model of brain neurons, were proposed more 30 years ago. But it was not until the breakthrough in 2006 that the rebranded “deep learning“ become popular. Beside algorithms, many factors collectively contribute to this breakthrough: computing power, large dataset and GPU.

The highlight of this book is that it really focus on a solid understand of the core concepts of neural networks. As he said,

Technologies come and go, but insight is forever.

on the exercises and problems

You should do most of the exercises because they’re basic checks that you’ve understood the material. If you can’t solve an exercise relatively easily, you’ve probably missed something fundamental.

The problems are another matter….With that said, I don’t recommend working through all the problems. What’s even better is to find your own project. Maybe you want to use neural nets to classify your music collection. Or to predict stock prices. Or whatever. But find a project you care about. Then you can ignore the problems in the book, or use them simply as inspiration for work on your own project. Struggling with a project you care about will teach you far more than working through any number of set problems. Emotional commitment is a key to achieving mastery.

1. using neural nets to recognize handwritten digits

2 important types of artificial neuron

perceptron

A perceptron takes several binary inputs, compare the weighted to a threshold value, produces a single binary output. So it’s a device that makes decisions by weighing up evidence:

$\sum_j w_j*x_j>threshold$

For convenience, vector and bias are introduced:

$\overrightarrow{w}\cdot\overrightarrow{x}+b>0$

In biological terms, if the condition is true, it is called fire.

sigmoid neuron

The output is modified by a sigmoid function (or logistic function), defined by

$\sigma(z)=\frac{1}{1+e^{-z}}$

so a large positive z lead to 1, and a large negative z lead to 0

If perceptron is regarded as a step function, then sigmoid neuron is a smoother version.

feedforward neural networks: out from one layer, used as input for next layer

stochastic gradient descent

cost function

$C(w,b)=\frac{1}{2n}\sum_x||y(x)-a||^2$

C is quadratic cost function or the mean squared error.

This cost function is more smooth than counting the correctly recognized numbers, because small changes to the weights and biases won’t change much.

This algorithm which make the cost as small as possible is called gradient descent.

The model is to imagine a ball rolling down to th ebottom of the vally (minimum C), with randomly choosing a starting point. The gradient vector is:

$\Delta C \approx \frac{\partial C}{\partial v_1} \Delta v_1 + \frac{\partial C}{\partial v_2} \Delta v_2\approx \nabla C \cdot \Delta v$

choose a proper learning rate $\eta>0$ and $\Delta v$ , to make a negative gradient:

$\Delta v = -\eta \nabla C$

Then the move of the ball is:

$v \rightarrow v' = v -\eta \nabla C$

The cost function is an average over individual cost, which can take a long time. To speed up learning, stochastic gradient descent is used, which chooses a small sample of randomly chosen training inputs (mini-batch).If only one training input is used, just as human being do, it’s called incremental learning.

code practice

The official MNIST data has 60 k tranning images and 10 k test images. Each training data is 784 dimensions of X and 10 dimensions of y (but the y value of test_data is not one hot encoded). So the number of neurons in the first layer is 784. mini-batch, e.g., each 10 training inputs as a group.

The following codes are run in Python command line. Bascially, it calls Network class from network.py, which defines a sigmoid function for feedforward and the derivative for back propogation.

import mnist_loader
training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
import network
net = network.Network([784, 30, 10])
net.SGD(training_data, 30, 10, 3.0, test_data=test_data)
# (training_data, epochs, mini_batch_size, eta)
#  score ~ 95%

starting point is much more important than algorithms and fine tune in parameters
If making a change improves things, try doing more!
Do you have enough training data?
Do you have enough epochs?
Do you have a good architecture?
Is the learning rate too low or too high?
sophisticated algorithm $$ \leq$$ simple learning algorithm + good training data
they’ve been learned automatically, so we understand neither the brain nor how AI works!
Deep neural networks (5~10 hidden layers) perform far better on many problems than shallow neural network(a single hidden layer).Because deep nets can build up a complex hierarchy of concepts.

Appendix: Is there a simple algorithm for intelligence?

The neural networks can be used to solve pattern recognition problems. The interesting question is will these approaches eventually be used to build thinking machines that match or surpass human intelligence?

Is there a simple set of principles to explain intelligence?

Intelligence may be explained by a large number of fundamentally distant mechanisms, which evolved in response to many different selection pressures in our species’ evolutionary history.

From connectomics, a brain contains 100 billion neurons with 100 trillion connections.If we need to understand the details of all those connections to understand how the brain works, we are not going to have a simple algorithm.

From molecular biology, human and chimp DNA differ at roughly 125 million DNA base pairs, out of a total of 3 billion DNA base pairs. So we are 96% chimp. If we converted these different genetic codes to letters, that’s about 30 times Bible.

Yet our genome alone is not enough to completely describe the neural connections, caveats are growing children need a healthy, stimulating environment and good nutrition to achieve their intelligent potential.

Chimp and human genetic lines diverged just 5 million years ago.

In 1980, Marvin Minsky developed “Society of Mind” theory, proposed that human intelligence is the result of a large society of individually simple computational processes:

What magical trick makes us intelligent? The trick is that there is no trick. The power of intelligence stems from our vast diversity, not from any single, perfect principle.

My own prejudice is in favor of there being a simple algorithm for intelligence. When it comes to research, an unjustified optimism is often more productive than a seemingly better-justified pessimism, for an optimist has the courage to set out and try new things. That’s the path to discovery, even if what is discovered is perhaps not what was originally hoped. A pessimist may be more “correct” in some narrow sense, but will discover less than the optimist.

That’s the path to insight, and by pursuing that path we may one day understand enough to write a longer program or build a more sophisticated network which does exhibit intelligence.

Sunday, October 23, 2016

What a data scientist is like and where they go?

A report by McKinsey Global Institute predicts by 2018 there will be about 1.5 million more jobs than skilled data scientists to fill those jobs. http://jychstar2.blogspot.com/2016/10/careers-in-data-science-nyu.html

Top 5 cities for big data jobs (June, 2014):

San Francisco
Mclean, Va.
Boston
St. Louis
Toronto

Big data professionals can be particularly difficult to find since many roles require a complicated blend of business, analytic, statistical and computer skills — which is not something a candidate acquires overnight. In addition, “clients are looking for people with a certain level of experience, who have worked in a big data environment. There aren’t a lot of them in the market

“What is this person going to be doing? Do you need the technical skills? Or is the quantitative/statistical expertise more important? Is this person going to be doing data modeling or making business decisions?” Kelley says. “In an ideal world, companies want all of it. But it’s not an ideal world.”

First data scientist at Pandora

2001, Gordon Rios was hired by Pandora as the first official data scientist. Three key lessons to take away.

The fully integrated Scientist

He’s completely fascinated with how people determine what to listen to, why and how their tastes and habits change.

He is a full-time member of a Playlist Team, which comes before he is a member of the data science team. He is embeded with a team of engineers, product managers, designers and others. Some companies have all of their DS sitting together, some have them working completely separately from the rest of the company; others follow a consultant-like model where DS parachute into projects temporarily.

Our mission is to make sure that artists to get listeners, and listeners to have the best experience possible.Both depend on getting people to try new music.And that depends on experimenting , collecting data and designing algorithms that push people outside of their musical comfort zones at the right pace.

You need people on operations, engineering, product, and scientists all coming at the problems from different sides, yet with a common vision for the service.

A consultant model would never work. When I first got started with data mining earlier in my career, I often worked as a consultant, and it’s very difficult to make progress on large-scale problems that way. You have to be part of the team to understand all the moving parts.

The best-case scenario is staffing data scientists that have good engineering skills. When DS can ship, you save on headcount and can turn data into meaningful products.

Rios brought ful-stack programming abilities, big data experience, and machine learning expertise to the table. He also had other critical skills that your first scientist needs: the abitliy to work autonomously, self-motivate and be accountable.

In most cases, good management is about lining up people’s skills to the company’s needs, but with data science, so much depends on having people be both skilled and interested.

The art of DS Management

“Of course there are times when you have to be a trooper and tackle a project
that is less than interesting yet critical to the company, but if you have
incredibly rich talent, matching them carefully with the right projects has
got to be what scientific management is all about,” Rios says. “Being able
to consistently do that separates good managers from okay managers.”

Even though they sit separately the majority of the time, the team holds regular
meetings and often rallies for lunch to talk about what they’re working on and
to have more informal conversations about ideas. A lot of solutions emerge outof these discussions. Slightly more formally, they schedule time to present
their projects and findings to their colleagues so they can ask or answer
questions, and/or share practices that might be helpful with other
experiments.

“Success for a data science program is when people are happy, feeling challenged and fulfilled, and delivering important results. That’s when they are at the highest performance state, delivering the most value,” says Rios. “There are a lot of reasons to bring on junior or inexperienced scientists —
they adapt and learn fast — but you better have really solid management and mentorship in place.”

“The hallmark of being a good manager or collaborator is that everyone
wants you involved in their project.” For the perspective employee, they have to bea culture fit firs and thena skill fit. They have to really love the product. and know about the data challenges we’re interested in solving.

Communicate for Optimum Efficiency

“To be an effective data scientist, you have to realize your job isn’t just about the research. You have to quantify and qualify what you do in a way that makes sense across the whole company.

“There are thousands of things we want to try, but we have to work
within the vision for what we want the service to be.”

Building and maintaining a sophisticated A/B testing platform is a must.

Make it standard to foster this transparency every time someone new comes aboard.

This is why we’re so focused on recruiting scientists who are scientifically curious but also entrepreneurial.

Friday, October 21, 2016

Head first C#

metacognition: trick your mind

your brain craves novelty. It’s always searching, scanning, waiting for something unusal.It was built to help you stay alive.
It doesn’t bother saving the boring things; they never make it pass the “this is obviously not important” filter.
How does your brain know what’s improtant. like tiger. like the danger of fire. what happens in your body? Neurons fire. Emotions crank up. Chemical surge. so your brain knows…
Imagine you are siting in a safe, warm, tiger-free zone. You are studying for an exam or trying to learn some tough technical topics which will take a week. Your brian is trying to do you a big favor. The brain’s scarce resources are better spent storing the really big things, not this obviously non-important, non-urgent content.
what turns your brain on? Make it visual. Use a conversational style. think more deeply. touch the emotions. ( you remember what you care about. You remember when you feel something)
the slow way to learn is sheer repetition. The faster way is to do anything that increase your brain activity, especiallly different types of brain activity.

visual application in 10 minutes

I use visual studio 2012, some buttons changes locations in different versions.

new project-> windows forms application, name it. you will get 3 .cs files, of which [design] is used as graphical coding;
toolbox -> pictureBox, drag and drop
rightup corner of the picture has a property arrow -> import image

in Form1.cs, add one line inside the Form1_Load method:

private void Form1_Load(object sender, EventArgs e)
    {
        MessageBox.Show("Contact List 1.0. \nWritten by: Yuchao", "About");
    }

debugging / stop debugging
In Solution Explorer, xx project->add -> New Item-> Local Database, name it
In Database Explorer, xx.sdf, Table ->creat table, set the first column as primary key and true identity, input the desired Database.
Project -> new data source -> database -> dataset -> next -> tables -> Finish
Datasource, drag and drop xx table, customize as you like.
project -> property ->publish -> publish now

.Net Framework

Buttons, checkboxes, lists, etc are all pieces of the .Net framework. It’s got tools to draw grpahiscs, read and write files, manage collections of things. The tools in the .Net framework are divided up into namespaces.

there’s nothing more attactive to a kid than a big sign that says, “Don’t touch this!” Come on, you know you’re tempted…

C# and .Net have lots of buit-in features
IDE chose a namespace for your code, e.g. your project’s namespace
your code is sotred in a class called Program
Every C# program must have exactly one “entry point” method that reads static void Main()
code snippet: for+table+table
when a new object is created from a class, it’s called an instance of that class
static method can be called without an instance
methods in a static class are all static
An object’s hehavior is defined by its methods, and it uses fields to keep track of its state.
Methods are what an object does.
Fields are what the object knows.

Due to time strain, I stop here. But I would definetely go back and spend one week to do the interesting excises in the remaining of the book.

Thursday, October 20, 2016

C++ refresher

/Volumes/BootCamp_Win7/_work/~All Codes/cpp work
/Volumes/BootCamp_Win7/_work/~OU Classes/Passed Classes

Brief history

Fortran, derived from Formula Translation, was invented in 1957 by John Backus at IBM around his 30s. “Much of my work has come from being lazy” during a 1979 interview. Fortran is the first high-level programming language and can perform relatively complex mathematical instructions such as in missile trajectories. Although old-fashioned, it is still used by physics and astronomy students due to many existing works.
C++ was invented in 1981 by Bjarne Stroustrup at Bell lab in his 30s.
Java was invented in 1995 by James Gosling at Sun/Oracle in his 40s.
Python was invented in 1991 by Guido van Rossum in his 35.

.h vs .cpp

Prefer to include the headers. They are C++ standard library headers. There are 18 <… .h> headers defined by the C standard library, but their use is deprecated.

.h: header file is for declarations. Declaration, such as int func(); tells the compiler what the class is, what data it holds, and what functions it has. It’s a interface telling user how to interact with the class.
.cpp: implementation file is for definitions, tells how the function works. The implementation details of the class are of concern only to the author of the class. Clients of the class don’t need to know and don’t care how the functions are implemented.

procedural/structured vs OO programming

structured programming: the principal idea is divide and conquer. Any task that is too complex to be described is broken down into manageable subtasks. This is very powerful. However, the deficiency is that the data structure is separated from functions and the reuse of function is difficult.
OO programming has 3 features: encapsulation (data hiding), inheritance, polymorphism, all saving you starting from scratch.

Compiling is like pre-cook food, so the compiled program can run very fast because the time-consuming task of translating the source code into machine language has already been done once.

what’s the problem I’m trying to solve?
Can this be accomplished without resorting to writing custom software?

resources

https://www.tutorialspoint.com/cplusplus/cpp_basic_syntax.htm

time-honored tradition

#include <iostream>

int main()
{
  std::cout << "Hello World!";
  return 0;
}

window disappear

Run or F5, the window disappear suddenly because the program finishes. Solutions:

Run without debugging (Ctrl+F5) ;
add char i; std::cin >> i for input
Debug->porperty->configuration properties ->Linker -> System ->SubSystem ->Console

Note: file input setting: Debug->porperty->configuration properties -> Debugging -> Command Arguments: “<input.txt”

comment

// line comment
/* block comment */

The guideline is that comments should say why it is happening stead of what it is happening.

to get rid of prefixing cout with std::, you can declare using namespace std;

Can I ignore warning messages from my compiler?

No! Get into the habit, from day one, of treating warning messages as errors.

Every C++ program has a main() function. When your program starts, main() is called automatically.

C-style string

char s[]={'H','e','l','l','o','\0'};
char t[]="Hello"

These two lines are equivalent. ‘\0’ is the null character, which is recognized as the terminator for a C-style string

string class

#include <string>
#include <iostream>
using namespace std;
int main()
{
  string s1("This is a C++ string!");
  cout<<s1<<endl;
  string s2=s1;
  cout<<s2<<endl;
  cout<<s1+s2<<endl;
  return 0;
}

function overload

They must differ in terms of parameter lists, either in the types or the number of inputs. The return type doesn’t matter.

pointer

int Age= 31; //
int *pAge=0; //  initialize pointer pAge to 0, a null pointer. Otherwise, a wild pointe is very dangerous
pAge= &Age; // addressing operator

int a[2]={1,2};
cout<< a <<endl;  // array name is actually a pointer

Animal *pDog=new Animal; // allocate memory
delete pDog;  // free the memory
pDog=0; // set pointer to null, avoid deleting a wild pointer

2 libraries

Because standard template library (STL) was developed before C++ was ISO standardized, there are some overlap between STL and C++ standard library. An example is you don’t need to include to perform the mathematical operation.

#include <iostream>
using namespace std;
const double pi=3.1415926535897;   //  not included in c++ library
int main()
{
  cout<<sqrt(2)<<endl;
  cout<<sin(pi/3)<<endl;
  cout<<exp(2)<<endl;
  cout<<log(exp(1))<<endl;
  cout<<log10(10)<<endl;
  cout<<pow(2,10)<<endl;

  return 0;
}

## Class and object
```cpp
// LISTING 10.4 - Using Constructors and Destructors
#include <iostream> // for cout

class Cat // begin declaration of the class
{
public: // begin public section
    Cat(int initialAge); // constructor
    ~Cat(); // destructor
    int GetAge(); // accessor function
    void SetAge(int age); // accessor function
    void Meow();
private: // begin private section
    int itsAge; // member variable
};

// constructor of Cat,
Cat::Cat(int initialAge)
{
    itsAge = initialAge;
}

Cat::~Cat() // destructor, takes no action
{
}

// GetAge, Public accessor function
// returns value of itsAge member
int Cat::GetAge()
{
    return itsAge;
}

// Definition of SetAge, public
// accessor function
void Cat::SetAge(int age)
{
    // set member variable itsAge to
    // value passed in by parameter age
    itsAge = age;
}

// definition of Meow method
// returns: void
// parameters: None
// action: Prints "meow" to screen
void Cat::Meow()
{
    std::cout << "Meow.\n";
}

// create a cat, set its age, have it
// meow, tell us its age, then meow again.
int main()
{
    Cat Frisky(5);
    Frisky.Meow();
    std::cout << "Frisky is a cat who is " ;
    std::cout << Frisky.GetAge() << " years old.\n";
    Frisky.Meow();
    Frisky.SetAge(7);
    std::cout << "Now Frisky is " ;
    std::cout << Frisky.GetAge() << " years old.\n";

    return 0;
}