Thursday, November 23, 2017

Robot ND, A4, Control, Deep Learning

I started a new job as Data Analyst this month. My journey in Udacity Nanodegree will pause for a while. Overall, the course material is of high quality and worth devoting more time to digest. Although I didn’t end up at exactly what I initially imagined, yet my mind has been significantly broadened with these up-to-date content.
This blog is a track record of how I made my career change during the past 18 months. Thank you, Blogspot.
There are too many things to say about deep learning. It is a black box. An accuracy below 99% is not that useful. It relies on huge labeled data, expensive GPU computing power, and complicated algorithms. Anyway, it is still in development and a lot of fun to learn!


Control engineering often referred to as simple controls, is a multidisciplinary topic with roots in engineering in applied mathematics.
The primary objective is to design systems so that the response to an input is a predictable and desirable output.
Virtually every organ in biological process in the body uses some form of control: body temperature, blood pressure, glucose, PH and even skeletal muscle reflexes all rely on feedback control systems.
two types of control:
  1. open loop control. no attempt is made to measure if the output actually is the desired response. wash machine. In early days, much of the control theory was developed for the chemical and material processing industries. It is better for highly predictable, non-safety critical system.
  2. closed loop control. Have a sensor to get feedback signal to the controller
97% of all regulatory controllers are PID.
PID is also called 3-knob controller: proportional, integral, derivative.

deep learning

  • Luis Serrano
  • Kelvin Lwin @ NVIDIA
fully convolutional networks:
  1. replace dense layer with 1*1 convolutional layer
  2. up-sampling through the use of transposed convolutional layers
  3. skip connection, use information from multiple resolution scales.
The Quiz uses tensorflow 0.12.1, as checked by tf.VERSION
object detection models that can draw bounding box: YOLO, SSD
semantic semmentation can acheive at pixel level.

segmentation lab

This ipynb contains the core codes for the final project “follow me”.
pip install tensorflow==1.2.1  # 1.1.0 doesn't work
pip install socketIO-client
pip install transforms3d
pip install PyQt5
pip install pyqtgraph
git clone
The train folder has 4132 images and corresponding masks, each image is shape (256, 256, 3)
The validation folder has 1185 images.
  • A lot of scaffold codes in the utils folder, including assigning images folder as X and mask folder as y. All the difficult details have been handled by utils codes. Don’t treat it as a magic button. Come back to dig more.
  • The images have been reduced to 128*128 to expedite the computing.
  • 3 classes are background, hero and other people.
A simple model looks like this:
def fcn_model(inputs, num_classes): 
    # Add Encoder Blocks, using separable convolution layers
    output1 = SeparableConv2DKeras (filters=3, kernel_size=3, strides=1, padding='same', activation='relu')(inputs)
    output1 = layers.BatchNormalization()(output1)    
    # Add 1x1 Convolution layer using conv2d_batchnorm().
    output2 = layers.Conv2D(filters=3, kernel_size=1, strides=1, padding='same', activation='relu')(output1)
    output2 = layers.BatchNormalization()(output2) 
    # Add the same number of Decoder Blocks as Encoder 
    upsampled = BilinearUpSampling2D((2,2))(output2)
    output3 = layers.concatenate([upsampled, inputs])
    return layers.Conv2D(num_classes, 3, activation='softmax', padding='same')(output3)


‘’ contains a lot of detailed instructions.

AWS setting

  • EC2
  • launch instance -> Community AMIs -> search Udacity robotics -> p2.xlarge,next, next, next,
  • security group. create new, source: my ip. Note: If you plan to connect a Jupyter notebook to your AWS instance you will need to add one connection rule. Specifically, you will need to add a custom TCP rule to allow port 8888. and set the source to My IP.
After launch
  • connect, pull out an instruction, refresh memory at
  • my key is stored at ~/.ssh
  • replace root in ssh -i "MyKeyPair.pem" to ubuntu. success.
  • connect AWS instance$ jupyter notebook --ip='*' --port=8888 --no-browser
  • open browse, enter {IPv4 Public IP}:8888
  • go back to terminal for token/password
  • if quit, type “exit” in terminal
  • if stop instance temporarily, click actions -> instance state -> stop

simulator control

L: Turns the legend with the control information on and off
H: Enables and disables local control of the quad
WSAD/Arrows: Moves the quad around when local control is on
E/Q: Rotate the quad clockwise and counterclockwise respectively
Space/C: Increase and decrease the thrust of the quad when local control is on
Scroll wheel: Zooms the camera in and out
Right mouse button (drag): Rotates the camera around the quad
Right mouse button (click): Resets the camera
Middle mouse button: Toggle patrol and follow mode
G: Reset the quad orientation
F5: Cycle quality settings
/: Path display on/off
ESC: Exit to the main menu
Crtl-Q: Quit

execute the code

cd RoboND-DeepLearning-Project/code
source activate RoboND
python  # image preprocessing
python my_amazing_model.h5 # test in simulator

What to include in your submission

Achieve an accuracy of 40% (0.40) using the Intersection over Union IoU metric which is final_grade_score at the bottom of your notebook.
Use the Project Rubric to review the project. You must submit your project as a zip file. The submission must included:
  1. Your model_training.ipynb notebook that you have filled out.
  2. A HTML version of your model_training.ipynb notebook.
  3. Writeup report (md or pdf file) summarizing why you made the choices you did in building the network.
  4. Your model and weights file in the .h5 file format

first submission review

The project code is basically not started and report only consists of a few lines, so the project cannot be reviewed. I think you would be better served discussing with a mentor on Live Chat rather than making a submission here.
As for how to improve your model, there are two main things:
1) Need to add more layers. At the moment there is just 1 encoder, 1 1x1 convolution and 1 decoder. You ideally want 2-3 encoders and 2-3 decoders.
2) Need to add more filters to each layer. At the moment just using 3 for each layer, but should be higher than this. For example 32, 64, 128, etc.
Again I highly recommend having a discussion with a Live Chat mentor before your next submission. We can help you out, don’t worry! :udacious: