Showing posts with label tool. Show all posts
Showing posts with label tool. Show all posts

Tuesday, September 26, 2017

Checklist Manifesto

The checklist manifesto: how to get things right. By Atul Gawande, 2009

0 introduction

a story of a stabbed man without knowing how deep the assailant’s knife went.
a story of a cancer operation when the patient’s heart suddenly stopped, due to the overdose of potassium by an anesthesiologist
As Goroviz and MacIntyre point out, we have just 2 reasons that we may nonetheless fail:
  1. ignorance. We only have partial understanding of the world
  2. ineptitude. though knowledge exists, yet we fail to apply it correctly.

1 The problem of extreme complexity

A story of a 3-year-old drowned girl, heart stop 2 hours before resuming, all organs recovered after 2 days except her brain. She went home after 2 weeks.
There is complexity upon complexity. And the answer is to go from specialization to super-specialization.

2 checklist

a story of Boeing Model 299 that is difficult to operate with too many details.
The vital signs every hospital records: body temperature, pulse, blood pressure, and respiratory rate.
In a complex environment, experts are up against 2 main difficulties:
  1. the fallibility of human memory and attention, especially when it comes to mundane, routine matters that are easily overlooked under the strain of more pressing events.
  2. people can lull themselves into skipping steps even when they remember them. In complex processes, after all, certain steps don’t always matter.
Checklists seem to provide protection against such failures. They remind us of the minimum necessary steps and make them explicit. They not only offer the possibility of verification but also instill a kind of discipline of higher performance.

3 The end of the master builder

Brenda (York Univ.) and Sholom (Univ. of Toronto) proposed 3 different kinds of problems in the world:
  1. simple: recipe, such as baking a cake
  2. complicated: like sending a rocket to the moon. Unanticipated difficulties are frequent.
  3. complex: like raising a child. Each child is unique. Although raising one child may provide experience, it does not guarantee success with the next child. The next child may require an entirely different approach from the previous one. The outcomes are highly uncertain.
A story of the construction of a skyscraper.
I saw a line-by-line, day-by-day listing of every building task that needed to be accomplished, in what order, and when. The schedule spread over multiple sheets. There was special color coding, with red items highlighting critical steps that had to be done before other steps could proceed.
submittal schedule: it is also a checklist, but it didn’t specify construction task; it specify communication tasks. The project managers dealt with the unexpected and the uncertain was by making sure the experts spoke to one another — on X data regarding Y process.
They don’t believe in the wisdom of the single individual, but in the wisdom of the group, the wisdom of making sure that multiple pairs of eyes were on a problem.
There’s program called ProjectCenter to deal with the tracking and communication
They trust in one set of checklists to make sure the simple steps are not missed or skipped and in another set to make sure that everyone talks through and resolves all the hard and unexpected problems.
The biggest cause of serious error in this business is a failure of communication.

4 The idea

2005 Katrina catastrophe.
The government did a poor job because it relies on a traditional command-and-control system which rapidly became overwhelmed. There are too many decisions to be made and too little information about precisely where and what help was needed. In face of an extraordinarily complex problem, power needed to be pushed out of the center as far as possible.
Walmart best recognized the complex nature of the circumstances. CEO Lee Scott said, “A lot of you are going to have to make decisions above your level. make the best decision that you can with the information that’s available to you at the time and do the right thing.”
Senior Walmart officials concentrated on setting goals, measuring progress, and maintaining communication lines. In a complex situation when conditions are too unpredictable and constantly changing, they did not issue instruction but worked on making sure people talked.
They had learned to codify that understanding into simple checklists.
Under conditions of complexity, not only are checklists a help, they are required for success.
Restaurant.
While Burger Kings and Taco Bells are driven by tightly prescribed protocol, in great restaurants the food is ever-evolving, refined, and individual.
Following the recipe is essential to making food of consistent quality over time.
In one case, a dish took too long for the eight-to-ten-minute turn-around that customers expect, so they re-engineered the dish, pre-prepared for the sauce and put into repeated test runs. the recipe was rewritten.
The chef also had developed a communication checklist to ensure people recognized and dealt with unexpected problems as a team. Half an hour before opening, everyone gathers in the kitchen for a quick check to discuss unanticipated issues and concerns, such as reservation count, menu changes, fill in for a sick staff member, and a party of 20 delayed into the dinner rush.

5 the first try

In 2006, a woman from world health organization request help to reduce avoidable deaths and harm from surgery. One of the benefits of joining up to work with WHO was gaining access to the health system reports and data from the organization’s 193 member countries. By 2004, surgeons were performing 230 million major operations annually.
In developing countries, the contaminated water is a source of many diseases. However, you have to wait decades for major infrastructure projects. So instead, they have to look for a low-tech solution.
Stephen Luby persuaded P&G to provide a grant for a proper study and to supply Safeguard soap with or without the antibacterial agent to a Pakistan neighborhood. They encourage people to use soap in 6 situations:
  1. wash bodies daily
  2. wash hands after they defecate
  3. wipe an infant
  4. before eat
  5. prepare food
  6. feed it to others
Each family received 3.3 bars of soap per week for one year. During this period, the incidence of diarrhea fell 52% no matter which soap was used. The secret is that the soap is more than soap. It is a behavior-changing delivery vehicle.
We want people to wash a lot. And people are quite poor. So we remove that as barrier.
The soap itself is also a factor. It smelled good and lathered better than the usual soap people bought. People liked washing with it. Global multinational corporations are really focused on having a good cosumer experience than public health people.

antibiotic

The step of applying antibiotic before an incision is commonly missed. Solution is:
  • a check box for the nurse to verbally confirm with the team
  • a metal tent was designed to cover a scalpel, serving as a reminder to run the checklist before making the incision.
  • It is also important to make clear that the surgeon could not start the operation until the nurse gave the okay and remove the tent. With this subtle cultural shift, even a modest checklist had the effect of distributing power.
There are 4 big killers in the surgery world: infection, bleeding, unsafe anesthesia and unexpected. A classic checklist could solve the first 3 except the last one. So they had determined that the most promising thing to do was just to have people stop and talk through the case together — to be ready as a team to identify and address each patient’s unique, potentially critical dangers.
Brian Sexton, a Johns Hopkins psychologist found that 1/4 surgeons believe that junior team members should not question the decisions of a senior practitioner. It turns out, the most common obstacle to effective teams is a kind of silent disengagement.
Their insistence that people talk to one another about each case, at least just for a minute before starting, was basically a strategy to foster teamwork.
Before starting an operation with a new team, there was a check to ensure everyone introduced themselves by name and role. Psychology studies show that people who don’t know one another’s names don’t work together nearly as well as those who do.
activation phenomenon: giving people a chance to say something at the start seemed to activate their sense of participation and responsibility and their willingness to speak up.

6 the checklist factory

Boeing example. fuel freezing to ice makes the plane lose power and crash.
Bad checklists are vague and imprecise.
Good checklists are efficient and easy to use even in the most difficult situations. They don’t try to spell out everything — a checklist cannot fly a plane. Instead, they provide reminders of only the most critical and important steps.
The checklist can’t be lengthy. A rule of thumb is 5~9 items, which is the limit of working memory. It depends on how many seconds you have.
Some use the checklist as it is, but many went on to make their own adjustments. Just as schools and hospitals tend to do things slightly differently, airlines are encouraged to modify the checklists to fit into their usual procedures.

7 The test

In aviation, “pilot not flying” starts the checklist. This is to spread responsibility and the power to question.
Using the checklist involved a major cultural change — a shift in authority, responsibility, and expectations about care.
Forcing the obstinate few to adopt the checklist might cause a backlash that would sour others on participating. We asked the leaders to present the checklist as simply a tool for people to try in hopes of improving their results.

8 The hero in the age of checklist

Just ticking boxes is not the ultimate goal, embracing a culture of teamwork and discipline is.
A story of investors. They don’t buy according to some computer algorithm. They do intensive research and invest for the long run.
Cook enumerated the errors known to occur at any point in the investment process — during the research phase, decision making, execution of the decision and even after making an investment. He then designed detailed checklists to avoid the errors, with clearly identified pause points to run through the items.
The checklist doesn’t tell him what to do, but help him be as smart as possible, ensuring he’s got the critical information, he’s systematic about decision making and he’s talked everyone he should.
Psychologist Geoff Smart studied how venture capitalists made their most difficult decision in judging whether to give an entrepreneur money or not. Finding an entrepreneur who can execute a good idea is more difficult than have a good idea. A person who can take an idea from proposal to reality, work the long hours, build a team, handle the pressures and setbacks, manage technical and people problems alike, and stick with the effort for years on end without getting distracted or going insane.
There are styles of thinking:
  • art critics. based on intuition and long-term experience
  • sponge. soaking every information then go with whatever their guts told them.
  • prosecutor. interrogate entrepreneurs aggressively, testing them with challenging questions
  • airline captains. methodical, checklist-driven approach. They forced themselves to be disciplined and not to skip steps.
Smart wrote a book, who, to explain this finding. But the major investor taking the checklist-driven approach doesn’t increase significantly.
We don’t like checklists. They’re not much fun. It is an embarrassment to our deep belief that the heroes improvise and handle situations of high stakes and complexity.
Hudson River Hero Sully Sullenberger III said it was crew effort. But we insist that’s just the modesty of the quiet hero. We don’t want to talk about teamwork and procedure. We want to talk about how great Sully is.
Professionalism has 4 common elements to expect:
  1. selflessness
  2. skill
  3. trustworthiness
  4. discipline
The most difficult is discipline. We are not built for discipline. We are built for novelty and excitement, not for careful attention to details. Discipline is something we have to work at.
Airlines put a publication date on all their checklists, because they are expected to change with time.
Technology can increase our capabilities, but it can’t deal with the unpredictable or manage uncertainty.
We’re obsessed with having great components, but pay little attention to how to make them fit together well.

Tuesday, March 14, 2017

Plotly is eating the data science market

Plotly is my new love after dimple failed to draw a simple pie chart. It turns out that, Plotly is doing the best job than any others that make every data analytics tool (e.g., python, R, Matlab) immediately ready for web-based data visualization. If you have complicated data and want to publish your finding online, look no further, python+ plotly is your best choice.
The founder of Plotly, Alex Johnson, has an interesting career path. He got his Harvard PhD in Physics in 2005, research on “Charge Sensing and Spin Dynamics in GaAs Quantum Dots” with a National Science Foundation graduate research fellowship. After one year of postdoc, he went to Harvard Environment Center for 3 years. He was trying to develop novel thin-film solid oxide fuel by applying semiconductor techniques such as microfabrication. He then spent 1 year at C12 Energy design and built a database and web interface for screening and forecasting enhanced oil recovery projects.
In 2012, he founded Plotly, which is a JavaScript graphing library:
  • comparable in scope and features to MATLAB or Python’s matplotlib.
  • It has D3 and WebGL for backend. no need for jQuery.
  • use JSON schema,It focuses on the chart’s physical attributes and attempts to leave the chart data separate.
  • In contrast, The vega and vega-lite schemas are more opinionated in prescribing how the chart data is grouped, sliced, or statistically processed before graphical display. This allows for complicated chart display with a concise JSON description, but leaves less control to the user. Neither approach is more “correct”—they’re just different.
Since 2015. 11, Plotly was open-source at https://github.com/plotly/plotly.js The business model for plotly is by charging the API of python. matlab, R, similar to the charge of Google map API. The community version is free for upto 50 API calls per day. More advanced plotting and more API calls are charged. Personal plan is $33 per month and student plan is $5 per month.
Due to time constraint, I only have a quick practice with the “getting started” for each language. Much more APIs are found here: https://plot.ly/api/

JavaScript

<script src=”https://cdn.plot.ly/plotly-latest.min.js”>
Basic Box plot
var y0,y1;
for (var i = 0; i < 50; i ++) {
    y0[i] = Math.random();
    y1[i] = Math.random() + 1;
}
var trace1 = {
  y: y0,
  type: 'box'
};
var trace2 = {
  y: y1,
  type: 'box'
};
Plotly.newPlot('myDiv', [trace1, trace2]);

python API

install the python API: pip install plotly
import plotly
plotly.tools.set_credentials_file(username='jychstar', api_key='1GPp9Dwmnsf897Z3kX8Q')  # now I use a free api key, info stored at .plotly/.credentials file in  home directory
plotly.tools.set_config_file(world_readable=False,
                             sharing='private') # public, private, or secret
import plotly.plotly as py
import plotly.graph_objs as go
trace0 = go.Scatter(
    x=[1, 2, 3, 4],
    y=[10, 15, 13, 17]
)
trace1 = go.Scatter(
    x=[1, 2, 3, 4],
    y=[16, 5, 11, 9]
)
figure = go.Figure(
    data = [trace0, trace1],
    layout = go.Layout (title = "hello world" )
)

py.plot(figure, filename = 'basic-line')  # plot online
py.iplot(figure, filename = 'basic-line') # plot inline
plotly.offline.plot(figure) # plot offline in browser
plotly.offline.iplot(figure) # plot offline in notebook
If you are interested what methods can be called by go, using dir(plotly.graph_objs)will give 58 items such as Area, Bar, Box, Candlestic,, HeatMmap, Histogram, Line, Pie, Scatter, Trace.
pie
labels=['Oxygen','Hydrogen','Carbon_Dioxide','Nitrogen']
values=[4500,2500,1053,500]
colors = ['#FAEE1C', '#F3558E', '#9C1DE7', '#581B98']
trace = go.Pie(labels=labels,values=values,marker={'colors': colors})
url_2 = py.plot([trace], filename='pie-for-dashboard', auto_open=True)
py.iplot([trace], filename='pie-for-dashboard')

pandas

use a fresh-bake model cufflinks , pandas can directly access plotly method.
import cufflinks as cf
import pandas as pd
cf.set_config_file(world_readable=True,offline=False)
pie=cf.datagen.pie()
pie.head()
pie.iplot(kind='pie',labels='labels',values='values')
df.iplot(kind='box', filename='box-plots')
df.iplot(kind='histogram', barmode='stack', bins=100, histnorm='probability', filename='histogram-binning')

matplotlib, seaborn

use existing plotting libraries, pass the handler to plotly.
import matplotlib.pyplot as plt
import numpy as np
import plotly.plotly as py
py.sign_in('jychstar', '1GPp9Dwmnsf897Z3kX8Q')
x= np.array(range(1,10))
y = np.sin(x)
fig, ax = plt.subplots()
ax.plot(x,y)
plot_url = py.plot_mpl(fig)  # publish fig online
## try seaborn, found the filling area is missing
import seaborn as sns
tips = sns.load_dataset("tips")
fig, ax = plt.subplots()
ax = sns.boxplot(x="day", y="total_bill", data=tips)
plot_url = py.plot_mpl(fig)

MatLab API

Download and uncompress the Plotly MATLAB library. 210K.
cd ~/Downloads/matlab_api/
plotlysetup('jychstar', '1GPp9Dwmnsf897Z3kX8Q')
[X,Y,Z] = peaks;
contour(X,Y,Z,20);
fig2plotly()  % push fig online
getplotlyoffline('https://cdn.plot.ly/plotly-latest.min.js') % download the offline plotly bundle
fig2plotly(gcf, 'offline', true) % generate html in current working directory
% try some simple plot
x= linspace(0, 5, 101);
plot(x, sin(x.^2), '-pb', 'linewidth',2)
fig2plotly()

R API

download latest R from CRAN
I happened to have a R version 3.3.1(2016.6.21). Because R 3.3.2 was release on 2016.10.31, I guess R API was released after that.
R.version  # check, plotly require R version 3.3.2
install.packages("ggplot2")  # plotly require version >2.1.0
install.packages("plotly")
library(ggplot2)
library(plotly)
Sys.setenv("plotly_username"="jychstar")
Sys.setenv("plotly_api_key"="1GPp9Dwmnsf897Z3kX8Q")
fig = plot_ly(midwest, x = ~percollege, color = ~state, type = "box")
plotly_POST(fig, filename = "midwest-boxplots")
# try to use other plotting library other than plotly
t = seq(0,10,0.1)
fig = qplot(t,sin(t),geom="path", xlab="time", ylab="Sine wave")
plotly_POST(fig, filename = "sine_wave")

Plotly vs Xap

Plotly Xap
web-based visualization yes yes
hosting final graph data pipeline
web presentation layout one full-size graph customized
plotting tool plotly.js + naive plotting library Component + JS library
generate take-away html, png files yes no
Python, R, Matlab works in naive environment yes no
open source yes ?
business model community/advanced consulting

Monday, July 25, 2016

Atom in my favor

Written with StackEdit.
I am not aware of the powerfulness of Atom until today.

Snippet

small prefix+ Tab= larger code block. Depending on the file type, snippet prefix include:
- html
- table
- img

favorite atom packages


  • open recent
  • autoclose-html
  • platformio-ide-terminal ( intergrate with terminal)
  • markdown pakages:
markdown-scroll-sync
linter-markdown
markdown-writer
markdown-toc
markdown-pdf
markdown-preview
markdown-preview-enhanced (interpret equation)

Sunday, July 24, 2016

Git & GitHub

git command

  • git clone [github link] [(optional)folder name]
  • git log
  • git log —stat
  • git diff first_id secod_id
  • git checkout
  • q
  • git config —global color.ui auto
    Atom as Git commit editor
    git config --global core.editor "atom --wait"

    make a repository

  • git init
  • git status
  • git add add new work to the staging area, a bridge b/w working directory and repository
  • git add . add all work
  • git commit will open a text editor, write, save and close editor.
  • git commit -m "message no need to open text editor
  • git diff (empty)
  • git diff --staged
  • git reset --hard
  • git branch
  • git branch easy-mode
  • git checkout easy-mode

    GitHub

  • git remote
  • git remote add origin url.git set the url as your remote reference
  • git remote -v stands for verbose
  • git push origin master sync to remote by master branch
  • git pull origin mastersync local by master branch
  • git config --global credential.helper osxkeychain password once for all
    -git pull=git fetch+git merge
  • pull request is actually merge request