Wednesday, August 10, 2016

IPND, stage 5, back-end, SQL

[TOC]

SQL

Structured query language
Database has several tables for different purposes.

basic query

select food from diet where species = "orangutan";
select * from tablename limit 10 offset 2;
select name, birthdate from animals where species != 'gorilla';
insert into tablename  values ( val1, val2, … );

select name from animals, diet where animals.species=diet.species and diet.food='fish';

select count (species),species from animals group by species order by count(species) DESC;
select animals group by species having num=1; # after aggregation

select ordernames.name, count(*) as num
  from animals, taxonomy, ordernames
  where animals.species = taxonomy.name and taxonomy.t_order = ordernames.t_order
  group by ordernames.name
  order by num desc;

update posts set content='cheese' where content like '%spam%';

delete from posts where content='cheese';

Create/drop database/table, primary key, references

create database name;
create table fish(id serial primary key, name text);
create table postal_places(postal_code text,country text, name text, primary key(postal_code, country));
create table sales(sku text references products, sale_date date, count integer); # use references for foreign key

drop database name;
drop table name;

delete from tablename;
insert into tablename values ('This is text!');

\c name # connect to database

join

select animals.name, animals.species, diet.food 
from animals join diet 
on animals.species = diet.speices
where food = 'fish';
-- show item with 0 count
select products.name, products.sku, count(sales.sku) as num
  from products left join sales
    on products.sku = sales.sku
  group by products.sku;
-- self join
select a.id, b.id, a.building, a.room
       from residences as a, residences as b
 where a.building = b.building
   and a.room = b.room
   and a.id > b.id
 order by a.building, a.room;

subQuery

select name, weight from players, (select avg(weight) as av from players) as subq where weight< av;
create view name as select ...

Python DB-API

use PosreSQL

always use 2 command windows

  1. for python
python forum.py
http://localhost:8000/ # you can open it on a browser
  1. for psql
psql forum   # load forum database, equal to" \i forum.sql"
select 2+2 as a, 4+4 as b;
select * from posts;
select * from posts \watch  # update very 2 seconds, so you can see what's added
\d posts  # get columns and type
\dt  # list all tables
\H #switch b/w plain text vs HTML

loop hole

something fun: http://xkcd.com/
sql injection attack: '); delete from posts; —
script injeciton attack

DB-API using sqlite3

import sqlite3
with sqlite3.connect("chinook.db") as conn:
    cursor = conn.cursor()
    rows = cursor.execute('select * from artist limit 10;').fetchall()
    conn.commit() # required if insert new data
for row in rows: print row
normalized design, which makes it easier to write effective code using a database.
A Simple Guide to Five Normal Forms in Relational Database Theory

normalized table

  1. Every row has the same number of columns,
  2. There’s a unique key. Everything in a row says something about the key.
  3. Facts that don’t relate to the key belong in different tables,
  4. Tables should’t imply relationships that don’t exist.

Saturday, August 6, 2016

IPND, stage 5, data analyst, notebook

update 2017-2-27

It is almost 7 months after the original post, which is really a mess from my current knowledge. I realize there is a huge cognitive gap between the learner and the teacher. This is where the “curse of knowledge” come in. Because we stand on the shoulder of giant. But the “giant” is hugely different for individuals. Different background, different culture, different learning pace, different learning style. We build new knowledge on top of what we already know. And the way we construct our knowledge is more like a Graph database. Knowledge is stored at the vertexes or edges. Our brain doesn’t store information in the SQL style.

Pandas

auto = pd.read_csv('data/auto.csv')
auto.head()
auto.describe()
auto.mpg.describe()
auto['mpg'].describe()
auto.mpg.std()
auto.price.hist()
auto.boxplot(column='price')
grouped = titanic.groupby('Sex')
grouped.Age.describe()

# add 2 pd.Series and fill missing value
s= s1.add(s2, fill_value = 0)
# def fun(x): return x**2
s. apply(fun)

df.loc[] # label based position
df.iloc[] # integer position
df.sort_values(ascending = False)

matplotlib

import matplotlib.pyplot as plt
plt.hist(list) or Series.hist()
x = np.arange(0, 5, 0.1)
y = np.sin(x)
plt.plot(x,y)
plt.xlabel()
plt.ylabel()
plt.title()
plt.show() #show plot in a new window

df.plot(kind='hist',title="passengers vs. sex")
plt.xlabel("gender, 0=male, 1=female")
plt.ylabel("number of passengers")

plt.legend(["dead","survived"])

seaborn

Seaborn is a Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics. Full customization of the figures will require a sophisticated understanding of matplotlib objects.
  • histogram
  • boxplot
  • kernel density estimation
  • violin plot
  • cumulative distribution function
import seaborn as sns
plt.hist(titanic.Age.dropna(),bins=25)
sns.boxplot(titanic.Age, titanic.Sex, vert=False)
sns.kdeplot(titanic.Age.dropna(), shade=True)
sns.distplot(titanic.Age.dropna())  # density + hist
sns.violinplot(titanic.Age.dropna()) # density + box
sns.kdeplot(titanic.Age.dropna(), cumulative=True)

Friday, August 5, 2016

IPND, stage 4, paths to choose

Basically, there are 4 paths for a programmer or developer to take:

1 front-end developer:

problems

responsive design
mockups to websites
fast loading speeds

different kinds

Design focus: HTML, CSS
Application focus: JavaScript

essential skills

Empathy: you aren’t developing a website for yourself, you’re creating it for other people
Breaking problems into manageable chunks.

frameworks

Angular.js, Ember.js, Backbone.js, Knockout.js, React.js and Polymer.js.

2 back-end programmer:

databases, deployment tools, back-end frameworks
The technologies used by back-end programmers tend to stick around longer than those used on the front-end because rearchitecting an application is a major undertaking. As a result back-end technologies change more slowly.
Languages like Java and Go are popular among large teams and large projects because they enforce code organization and structure. Languages like Python, Ruby and Javascript allow rapid development and easy prototyping, but they don’t necessarily scale to heavy loads as easily. This implies cultural differences between the communities around different languages, and in the teams that use the different languages. Java in particular is used at larger enterprises. PHP tends to be seen in legacy applications.

3 Mobile programmer: iOS, Android

problems

responsive design
clean user interface
sync to server
from a functional state to a production ready state
mobile hardware

essential skills

oo programming, networking (online APIs)
programming paradigm called model view controller
ios: swift (since 2014 summer), objective C, IDE is Xcode
android: Java, IDE is Android Studio

4 data analyst.

Python for data wrangling, D3. js for data visualization, R for statistical analysis

problems

trends from data
applied statistics, machine learning
data visualization
Jessica Kirkpatrick, data scientist at hired, PhD at astrophysics.
In her view, data analysts is to basically help other people in the company make decisions and prioritize their work by using the data we collect
language: sequel to fetch data, or similar language (Hadoop or MapReduce), statistical package like R, python, whatever for more advanced statistics and modeling and numerical analysis

essential skills

  • Mathematics and statistics. Understand how to interpret the results, the technical aspects of models that are employed, develop new or alternate techniques to go beyond the built-in.
  • Programming. Beyond software package to perform “out of box”. customized effects
  • a sense of curiosity. Ask questions of their data to generate a logical flow in analysis. If an oddity is detected, such as missing data, outliers, unexpected trends, steps should be taken to understand the oddity and try to resolve it.

python vs R

R. Best at very specialized task:

  1. ggplot2: good-looking visualizations to hasten exploration of the data
  2. dplyr, tidyr: reshaping data

    Python. easier to learn and understand

  • scikit-learn, matplotlib, seaborn: for machine learning and visualization

learning zone:

safe-stretch-strain
The Safe Zone - This is where concepts are familiar and we don’t have to take any risks. The Safe Zone is important because it gives us a place to return and reflect on our current understanding. However, our potential for growth is limited in the Safe Zone because we are not challenged with unfamiliar concepts.
The Stretch Zone- To grow we must leave our Safe Zone and explore a Learning Zone just beyond our secure environment. Only in the Learning Zone can we make new discoveries and slowly expand our Safe Zone. Entering the Learning Zone is a borderline experience. We feel we’re exploring the edge of our abilities and our limits.
The Strain Zone- Beyond our Stretch Zone is the Strain Zone where learning becomes difficult and is blocked by a sense of fear and frustration. Learning that is connected with negative emotions is memorized in a part of the human brain that we can access only in similar emotional situations. While learning is possible in the Strain Zone, most of our energy is used managing and controlling our anxiety.
dancer, psychologist and coder: http://georgiatdavis.com/

Thursday, August 4, 2016

Unitarian Universalist

I happened to know UU last week in my previous post. And I paid a real visit to the First Unitarian Church of Oklahoma City last Sunday. Here were my encounters:
  1. We got a warm welcome there and a lady showed us around, handed us booklets, and introduced us to everyone we met.
  2. The number of people there is smaller than I expected. Just a few tens. Maybe it’s because summer session.
  3. To eat a donut, you will have to pay. But the coffee is free.
  4. The order of service at 11 AM took place at Sanctuary. They usually take turns to lecture. Today is about UU jokes, which totally blew my mind. I am surprised they are so open to different beliefs and opinions that they can laugh at themselves.
Here are my favourite UU jokes:
  • What do UUs have in common with Pontius Pilate?
    They ask, “What is truth?” and then don’t stay around for an answer.
  • What’s the difference between an agnostic, an atheist, and a Unitarian?
    I don’t know, and I don’t care one way or the other.

A youtube video by Aaron White explains the basics:

  1. rejection the notion of Trinity. Jesus is a human prophet.
  2. rejection the notion of original sin.
  3. rejection the notion of eternal punishment. How could God do that to God’s children?
    book: a treatise on atonement.
    It makes no sense to me that an infinite God infinitely punish a finite human being for making finite mistakes. An unlimited force brought down in punishment unlimited to people who do what limited people do.
We take the scripture and traditions of our Christain ancestors seriously, but not literally.
Credo religion vs Covenantal religion. we are the latter kind. I don’t come here to be better or more worthy than anyone else or to secure a special place for myself in my family. Everyday we wake up and keep a promise we’ve made about how we will live.
A promise guided by profound humility and active love.
  1. even we want certainty about life’s biggest questions,we’re unlikely to find it.
  2. not passive response
As limited human beings, we just can’t know everything. Our best response to the gift of life is to love and serve as many others as we can while we’re still alive for this might be the only life we have.
whether the story is true or not is not important; but what’s true about it?

The Five Reasons You Shouldn’t Be a Unitarian

  • you can’t believe anything you want.
  • I don’t have argument with fundamentalism, because I can’t argue with them.

My local branch:

West Wind UU
1309 W Boyd St, Norman, OK 73069, United States
Sundar Flansburg on 2016-7-10
-what disturbs me: so many people seem to blindly apply their very limited experiences to other people and complex societal problems, failing to recognize that those experiences are not shared ones. Raising children, getting a job, driving through the city, going out to a club to dance—all of these things can be profoundly different experiences, depending on your race, gender, sexual identity, socioeconomic status. We need life skill, community building skill, of being able to step outside of our direct knowledge and imagine something else.
  • I’m convinced that in sharing stories we develop and expand empathy Like travel, art, music, reading, and other ways we step out of our everyday lives, listening deeply to other stories helps us see the world in a different way, helps us step into the experiences of people different from us.
Maureen Harvey on 2016-2-14
  • Fear stands in our way of living in a state of pure love. It causes us to be small, to shrink, to disconnect, to isolate. So, what can we do about fear? I believe it’s best to invite fear in and allow ourselves to really feel it, to sit with it, of course, draw about it, befriend it and learn from it, including our fear of dying. Resisting unpleasant feelings like fear causes them to grow and wreck havoc in our lives while befriending such feelings helps them to shrink.
  • Study shows 85% of the things we worry about never happen, and when things did happen, most people discovered they could handle the difficulty better than they expected, or they learned a valuable lesson from it.
    -97% of what you worry over is not much more than a fearful mind punishing you with exaggerations and misperceptions.
  • An experience we had 10 years ago does not have to define us. We are not the same as we were last week or yesterday. Our feelings are just feelings and our thoughts are just thoughts—they are not who we are.
  • If each one of us focuses on one issue we are passionate about, and puts our energy towards justice in that area we can make big changes.

claims

  • God is not so much about what I believe. God is about what and how I serve.
  • We believe that personal experience, conscience, and reason should be the final authorities in religion. We put religious insights to the test of our hearts and minds
  • We believe that religious wisdom is ever changing. Human understanding of life and death, the world and its mysteries, is never final. Revelation is continuous.
  • We affirm the worth of every person. We believe people should be encouraged to think for themselves. We know people differ in their opinions, choices, and identities, and we believe these differences should be honored.

resources, books

notable believer

  • John Adams(2nd president of US): regular church service was beneficial to man’s moral senses; strove for a religion based on a common sense sort of reasonableness;
  • Clara Barton (founder of US red cross)
  • Oliver Wendell Holmes Jr. (Asso.Justice of Supreme Court)
  • Louisa May Alcott (author of little woman)
  • Waldo Emerson (thinker)

Matt’s Tech Conference Survival Tips

  • Today’s networking contacts are tomorrow’s clients
  • If you’re the smartest person in the room, you’re in the wrong room.
  • ‘Don’t take on any project unless you’ll learn something from it’. So my take away here is when given a choice, choose the topic that pushes your boundaries, even if some of it goes over your head.
  • Kick it Old School – Use Paper notebook & pens. maximize your focus.