Theano with GPU, setup and trouble shooting

Environment setup is always a headache for me. The only way to fight back is documenting every step I did before as detail as possible.

I will find time to finish this udacity course on Linux Comand line

I am now using MacBook Pro (2013) with OS X El Capitan, 10.11.6.

cpu: 2.4 GHz i7;

memory: 8G

graphics : NVIDIA GeForce GT 650M 1024 MB.

[TOC]

Background knowledge refresh

To have effect after making some changes, you need to restart terminal.Alternatively. you could execute source ~/.bash_profile to reload your settings.

Bash file

There may or may be not a default file in home directory named .bash_profile.

I configure the git workspace by this udacity video about tab completion, prompt color and git editor. Because I ran git config --global core.editor "atom —wait", so each time if I git commit and open a new file to documment the changes, I have to close the file before I continue.

Set path

to check your working directory: pwd

to check path: echo $PATH

The paths shown are actually written in two files:

/etc/paths

~/.bash_profile

in the first file, you can direct write the path; in the second file, you write your path in the format

export PATH="xxx:$PATH"

xxx represent your actual path.

The nice thing about bash_profile is you can use alias mlnd=" cd Desktop/Udacity/MLND/course_material/projects/" to save a long path for your frequent use.

Theano

Theano is a machine learning library, http://deeplearning.net/software/theano/

• easy to implement backpropagation for convolutional neural networks
• can run code on either a CPU or a GPU.

Basically, there are 3 steps to use Theano with GPU enabled:

1. sudo pip install Theano
2. download and install CUDA toolkit,which includes a compiler for NVIDIA GPUs. https://developer.nvidia.com/cuda-toolkit.
3. configure Theano and CUDA

enable GPU

after nosetests theano, I found something is missing, then pip install nose_parameterized

write these into .bash_profile

# Theano and CUDA
export PATH="/Developer/NVIDIA/CUDA-8.0/bin/:$PATH"
export LD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-8.0/lib/
export CUDA_ROOT=/Developer/NVIDIA/CUDA-8.0/
export THEANO_FLAGS='mode=FAST_RUN,device=gpu,floatX=float32'

test GPU or cpu

from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time

vlen = 10 * 30 * 768  # 10 x #cores x # threads per core
iters = 1000

rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in range(iters):
    r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
    print('Used the cpu')
else:
    print('Used the gpu')

used the cpu took 2 .06 seconds; used the gpu took 1.22 seconds

Another guide

http://daoyuan.li/installing-theano-and-cuda-on-mac-os-x/ provides detailed guidance and 2 comparison examples of GPU/CPU

write a python file, check.py

from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time

vlen = 10 * 30 * 768  # 10 x #cores x # threads per core
iters = 1000

rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in xrange(iters):
    r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
    print('Used the cpu')
else:
    print('Used the gpu')

in command line:

THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32 time python check.py 
THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 time python check.py

The results is 8.44 vs 6.3

But another test file, lr.py shows the opposite.

I also try example codes of chapter 6 from “Neural networks and deep learning” by Michael Nielsen. I get 60% reduction of running time. However, my excitement didn’t last long. I soon get “Initialisation of device gpu failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY”. I tried to set cnmem= 1.0 or 0.9, but no help.

At last, I resort to https://github.com/phvu/cuda-smi to check my gpu. Still no idea.

I had to close everything and restart. still failed! Then I change cnmem =0.1 and succeed. What a joke~ I realize my poor GPU is not meant for such deep learning computing. The memory management is far from optimization. Remember: theano is at 0.8 version, CUDA is also at 0.8 version. They are still at very early stage. Note that nvidia just built first AI supercomputer, which cost $129 k!

life is better with cloud service

http://www.pyimagesearch.com/2014/10/06/experience-cudamat-deep-belief-networks-python/ didn’t get an improvement after using GPU, and suggested using cloud service:

http://www.pyimagesearch.com/2014/10/13/deep-learning-amazon-ec2-gpu-python-nolearn/

​

Fundamental of machine learning for predictive data analytics

by John D. Kelleher 2015

Best book in machine learning I have read so far! Cover very practical perspectives of machine learning problem.

1 Introduction

Because of the noise nature and finite sampling, Machine learning is an ill-posed problem, which can’t be completely determined by a unique solution.

In fact, searching for predictive models that are consistent with the dataset is equivalent to just memorizing the dataset. As a result, no learning is taking place it tells us nothing about the underlying relationship between the descriptive and target features. If a predictive model captures this underlying relationship between the descriptive and target features, it is said to generalize well. The goal of machine learning is to find the one generalizes best.

Machine learning is sometimes called inductive learning, because it learns a general rule from a finite set of examples. Every machine learning algorithm has inductive bias, which is a set of assumptions. Two types of inductive bias are restriction bias and preference bias. A inductive bias is a necessary prerequisite for learning to occur; without inductive bias, a machine learning algorithm cannot learn anything beyond what is in the data.

No particular inductive bias on average is the best one to use. (No Free Lunch Theorem). A core skill for a data analyst is to select the appropriate model. An inappropriate inductive bias can lead to underfitting or overfitting, when the model is too simple or too complex. The goal is to strike a good balance.

CRISPDM: cross industry standard process for data mining:

• Business understanding. The goal of predictive data analytics projects is not building a prediction model, but things like gaining new customers, selling more products, or adding efficiencies ot a process. So, during the first phase in any analytics project, **the primary goal of the data analyst is to fully understand the business or organizational problem that is being addressed, and then to design a data analytics solution for it.
• Data Understanding
• Data preparation. convert required data sources into a well-form analytics base table(ABT)
• Modeling
• Evaluation
• Deployment

predictive data analytics tools

• application-based solution: IBM SPSS, SAS
• programming. This has more flexibilities and newest analytics techniques, but learning curve is steeper and need to put extra burden on developers to implement infrastructural support as data management.

2 Data to insights to decisions

Albert Einstein

We cannot solve our problems with the same thinking we used when we created them

Organizations don’t exist to do predictive data analytics. Organizations exist to do things like make more money, gain new customers, sell more product or reduce loss. The prediction don’t solve business problems, but provide insight that help the organization make better decision to solve their business problem.

converting a business problem into an analytics solution:

1. What’s the business problem? What are the goals that the business wants to achieve? Most of the time, organizations begin analytics projects because they have a clear issue that they want to address; but sometimes it’s simply because somebody in the organization feels that this is an important new technique that they should use it. Unless a project is focused on clearly stated goals, it is unlikely to be successful
2. How does the business currently work? It’s not feasible for an analytics practitioner to learn everything about the business because they will move quickly between different areas. But they must possess situational fluency, that they can use correct terminology to build analytics solution for that domain.

data availability

A lack of appropriate data will simply rule out proposed analytics solutions to a business problem. The easy availability of data for some solutions might favor them over others.

4 Information-based learning

decision tree

Model Ensembles

Rather than creating a single model, they generate a set of models and then make predictions by aggregating(such as voting) the outputs of these models. A prediction model that is composed of a set of models is called a model ensembles.

Two standard approaches:

• boosting. works by iterating creating models, which add biased to pay more attention to instanced misclassified by last model.
• bagging(or bootstrap aggregating). Each training data set has a random sampling with replacement. A decision tree bagging and subspace sampling is called random forest.

5 Similarity-based learning

nearest neighbor algorithm is a lazy learner, which delays abstracting from the data until it is asked to make a prediction. It is relatively slow because it needs to store a large number of instances. It’s sensitive to redundant and irrelevant descriptive features. The advantage is that it’s robust to concept drift, which means the relationship between features and target may change over time.

6 Probability-based learning

Bayes’ Theorem

As early as 1700, Reverend Thomas Rayes wrote an essay that described how to update beliefs as new information arises. The modern mathematical form was developed by Laplace.

If X is target, Y is features, then Bayes’ Theorem can be used for prediction. Take X for categorical values, the maximum probability of is chosen as the predicted value.

Prediction is kind of inverse reasoning (from evidence to event), which is often much more difficult than forward reasoning (from event to evidence). 事后诸葛亮. 20/20 hindsight.

Naive Bayes model is naive because it simply assume the independence between features. This greatly reduces the difficulty of computation. The maximum mechanism make it robust to noise.

7 Error-based learning

The value chosen for the learning rate and initial weights can have a significant impact on how the (batch) gradient descent algorithm proceeds. However, how to choose is more an art gathered through experience, rather than a well-defined science.

The gradient descent algorithm requires the results to be differentiable, but the simple linear regression with sign function fails to that, so the logistic regression with the logistic function cuts in and do the job:

where x is the weighted sum.

Support vector machine took another approach, where the margins are defined by the support vectors. The negative target feature is set to -1 and the positive target feature is set to +1. Kernel trick is played on the descriptive features to moves the data into a higher-dimensional space.

11 The art of ML for predictive data analytics

Sherlock Holmes

It is a captial mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.

Predictive data analytics projects use machine learning to build models that capture the relationships in large datasets between descriptive features and a target feature. Machine learning is a type of inductive learning, so they share some properties:

1. the general rule induced from a sample may not be true for all instances in a population
2. learning cannot occur unless the learning process is biased in some way, which means we need to tell the learning process what types of patterns to look for in the data. This bias is referred to as inductive bias.
3. the outcome is also intentionally biased to suit our need.

An analytics project is often iterative, with different stages of the project feeding back into later cycle. It is also important to remember that the purpose of an analytics project is to solve a real-world problem and to keep focus on this, rather than being distracted by the admittedly sometimes fascinating, technical challenges of model building. The best way to keep an analytics project focused and to improve the likelihood of a successful conclusion, isto adopt a structured project lifecycle like CRISP_DM.

choosing a machine learning approach

No free lunch theorem.

A simple example shown in figure 11.2 reveals that each ml algorithm has its edge. The decision boundaries learned by each algorithm are characteristic of that algorithm

For small dataset, generative models are preferred than discriminative models because the prior structural information is encoded into the generative models, which can be used to generate data.

​

Machine learning ND 3, reinforcement learning

Reinforcement learning

This project is easier than I expect. It only use very simplied version of Bellman equation. Use a dictionary to implement the Q-learning greatly reduces the complexity.

Smartcab

cd Desktop/Udacity/MLND/course_material/projects/smartcab/

pygame

pip install pygame
python smartcab/agent.py

warning: don’t use conda install! It wasted me 2 hours to realize what’s the problem.

With GUI open, a single trial has about 120 steps and takes about 4.5 minutes.

code structure

agent.py{
  class LearningAgent(env.Agent):{
    __int__(env,learning,epsilon,alpha)
    reset()
    build_state()
    get_maxQ(state)
    createQ(state)
    choose_action(state)
    learn(state,action,reward)
    update()
  }
  run()
}

simulator.py{
  class Simulator(){
      __init__(env,size,update_delay,
      display,log_metrics,optimiazed){
      / line 90-110 write header
      }
      run(tolerance,n_test){
      /line 133 "total_trials>20"for training number control
      /line 229-245 write data to csv file
      }
      render_text(trial,testing){}
      render(trial,testing){}
      pause(){}

  }
}

environment.py{
  class TrafficLight(){}
  class Environment(){
    __init__(verbose,num_dummies,grid_size){}
    create_agent(agent_class,*args,**kwargs){}
    set_primary_agent(agent,enforce_deadline){}
    reset(testing){}
    step(){}
    sense(agent){}
    get_deadline(agent){}
    act(agent,action){}
    compute_dist(a,b){}
   class Agent(){}
   class DummyAgent(Agent){}
  }
}

key implementation:

with GUI off, this runs super fast.

revise run() by:

env = Environment(verbose = False, num_dummies = 100, grid_size = (8, 6))
agent = env.create_agent(LearningAgent, learning = True, epsilon = 1,  alpha = 0.3)
env.set_primary_agent(agent, enforce_deadline = True)
sim = Simulator(env, size = None, update_delay = 0.01, display = False, log_metrics = True, optimized = True)
sim.run(tolerance = 0.05, n_test = 10)

args and *kwargs in python

implement function by:

def reset(self, destination=None, testing=False):
    self.planner.route_to(destination)
    #self.epsilon -= 0.05  # decaying function for question 6
    self.epsilon *= 0.95 # for question 7
    if testing:
        self.epsilon, self.alpha = 0, 0
    return None

def build_state(self):
    waypoint = self.planner.next_waypoint() 
    inputs = self.env.sense(self)          
    deadline = self.env.get_deadline(self) 
    state = (waypoint,tuple([inputs[item] for item in inputs]))
    return state

def get_maxQ(self, state):
    maxQ = float('-inf')
    for key,value in self.Q[state].iteritems():
        maxQ = max(maxQ, value)
    return maxQ

def createQ(self, state):
    if not self.Q or state not in self.Q:
        self.Q[state] ={None:0.01,   'left':0.0, 'right':0.0,    'forward':0.0}  # give idle a slightly priority
    return

def choose_action(self, state):
    self.state = state
    self.next_waypoint = self.planner.next_waypoint()
    # action = None
    waypoint = self.next_waypoint
    import random
    actions = [None, 'left','right', 'forward']  

    if not self.learning:
        action = random.choice(actions)
    else:
        highest = self.get_maxQ(state)
        action_dict = self.Q[state]

        coin = random.random()
        if coin <= self.epsilon:
            if action_dict[waypoint] == 0.0:
                action = waypoint
            else:
                action = random.choice(actions)
        else:
            for key,value in action_dict.iteritems():
                if value == highest:
                    return key
    return action

def learn(self, state, action, reward):
    if self.learning:
        self.Q[state][action] += self.alpha*reward   
    return

​

Machine learning ND 1, supervised learning, material collection

the original post has merged into github: https://github.com/jychstar/NanoDegreeProject/tree/master/MachineND/p2_Finding%20Donors

​

Religion for the nonreligious

Original article by Tim Urban. Here are my excerpt

Society at large focuses on shallow things that it doesn’t stress the need to take real growth seriously. Religions make salvation the end goal instead of self-improvement.

Hundreds of millions of years of evolution has created a zoo of small-minded emotions and motivations in our heads: fear, pettiness, jealousy, greed, instant-gratification.

Over the past 6 million years, human have taken a big step up the consciousness staircase, which we can call “Higher being” . It’s brilliant, big-thinking and rational, but he’s a very new resident in our heads.

So the human brain is a strange world: a combination of the Higher being and the low-level animals.

It was not until the higher being realized “we’re going to die“ that he can take over the control over other animals.

The collective force of the animals is what I call “the fog“. The more the animals are running the show and making us deaf and blind to the thoughts and insights of the Higher Being, the thicker fog is around our head, which make us only see a few inches in front of us.

so our problem : the battle of the Higher Being against the animals is the core internal human struggle: try to see through the fog to clarity

Typical struggles:

• Rational Decision Maker vs. Instant Gratification Monkey
• Authentic Voice vs. Social Survival Mammoth

The difficult thing is : when you are in the fog, you don’t know you are in the fog. The thickest fog, the less you are aware of its existence. So the key is being aware that the fog exists and learning how to recognize it.

step 1: our lives in the fog

Higher being: high-minded, love-based, advanced emotions

brain animals: small-minded, fear-based, primitive emotions

tribalism makes us hate people different than us

The power of fog

• bend and loosen your integrity, for tiny insignificant gains which affect nothing in the long term
• let the fear of what others might think dictate the way you live, but actually everyone is buried in their own lives without really thinking about you anyway.
• keep them in the wrong relationship, job, city, apartment, friendship.
• promise fake future happiness which fade away quickly due to Hedonic Treadmill. The fog itself is the source of unhappiness.

step 2: thinning the fog to reveal context

• broaden perspectives: education, travel, life experience
• active reflection: journal, therapy, ask questions like “what would I do if money is no problem”, “how would I advise someone else on this”, “will I regret not doing this when I’m 80? “ ask your Higher Being’s opinion on something without the animals realizing what’s going on. These tricks help the animals stay calm so the Higher Being can actually talk.
• Activities that help quiet brain’s unconscious chatter: meditation, exercise, yoga

Most easiest way to thin out the fog is be aware of it. look at the whole context keeps you conscious, aware of reality.

examples:

The cashier is rude to me.

This dude’s in a dark place. Maybe his day has sucked.

life is so unfair.

Look at so many things I have.

Everything is amazing forever.

It’s part of a rocking curve.

Life is bad from here forward.

It’s part of a rocking curve.

Everything is scary. why did I say that? why am I so embarrassing?

Sometimes human brains great out and think these shitty things. I can feel my brain doing that right now. Oh brains

I don’t know what’s going on. I will get rid of these weird things.

I see the consequences and I will take actions.

When we’re on step 2, this broader scope and increased clarity makes us feel calmer and less fearful of things that aren’t actually scary. and the animal who gain strength form fear become ridiculous.

It’s extremely hard to stay on step 2 for long. But you can get better at noticing when it’s thick and develop effective strategies for thinning it out.

step 3 shocking reality

Our brain can’t handle the vastness of space, eternity of time, or the tininess of atoms. You can do it if you focus, but it’s a strain and you can’t hold it for very long.

A whoa moment is like being at the Grand Canyon. It’s difficult to maintain for very long, but only in a whoa moment does your brain actually wrap itself around true reality.

They make me feel ridiculously, profoundly humble. In those moments, all those words religious people use — awe, worship, miracle, eternal connection— make perfect sense. I want to get on my knees and surrender. This is when I feel spiritual.

In those fleeting moments, there’s no fog. My Higher being is in full flow and can see everything in perfect clarity. The animals become the sad little creatures with no fog to obscure things.

Each time you humiliate the animals, a little bit of their future power over you is diminished.

From Step 1 view to see an atheist, life on Earth is taken for granted.

From step 3’ view, life itself is more than enough to make me excited ,lucky and loving. How cool it is that I’m a group of atoms that can think about atoms.

Step 4, the great unknown

Carl Sagan:

science is not only compatible with spirituality, it is a profound source of spirituality.

If we ever reach the point where we think we thoroughly understand who we are and where we came from, we will have failed.

What if a more intelligent species tried its hardest to explain something to us? We may be not able to grasp anything.

Remember the powerful humility mentioned in step 3? Multiply by 100. That’s step 4. If I’m just a molecule floating around an ocean I can’t understand, I might as well just enjoy it.

what next?

nothing clears fog like a deathbed.

There are plenty of good things in the religious world, but it’s something happening in spite of religion and not because of it.

A Truthism knows:

• where to put my focus
• what to be wary of
• how to evaluate my progress, which will help me make sure I’m actually improving and lead to quicker growth

several questions:

• what’s the goal that you want to evolve towards? why that goal?
• what does the path look like that gets you there?
• what’s in your way? How do you overcome those obstacles?
• what are your practices on day-to-day level?
• what should your progress look like year-to-year?
• Most importantly, how do you stay strong and maintain the practice for years and years, not four days?

Articulate it helps clarify it in your head.

​