A hands-on tutorial for Xap beginners

Not every internet product survives in today’s fierce competition. I believe the survival of Exaptive platform in the next 5 years will require a design paradigm shift. This tutorial is not only for Xap users, but also for Xap developers. The overarching theme is building a minimum viable product (MVP). With such a skeleton at hand, users will have a lot of fun by tinkering here and there.

Data scientists come from different backgrounds and have their own preferred technology stacks. Exaptive platform aims to bridge the gap between offline data visualization and online data visualization. The latter means web-based, which leverage the power of fast-growing JavaScript libraries such as D3, Dimple and Plotly.

To positioning this tutorial, the targeted learners in my mind are frequent Python users, who are already familiar with data structure like dictionary, and module like pandas. From my learning experience, there are 2 major knowledge gaps:

1. HTML/JavaScript
2. Data types and conversion in Exaptive environment

HTML/JavaScript

These are prerequisites for any web applications that are tailored to exactly what you need. It is easy to learn because you can practice it in the browser and get immediate feedback. You can learn all the fundamentals at https://www.w3schools.com/. My learning note is here.

Once you know how to control the DOM by JavaScript, and how to use jQuery or D3 to draw a simple shape, you are good to go.

Data types and conversion in Exaptive

This is the most confusing and buggy thing for a beginner in Exaptive. I will provide some easy-to-follow steps to walk you through the mist. I am going to open every black box to help you get a clear understanding of the magic.

Get familiar with the platform

Open https://exaptive.city/, there are 4 options at the top:

• Home. This is the warehouse for storing your Xap.
• Studio. This is the warehouse for storing your Xap and Components. If you click + button, there pops up there things: Xap, Component, Asset. Component is the basic building block. Xap is the product when you piece required components together. Asset is a dataset, code snippet, pictures that are used to build a component or feed data. The current version of Studio is 4.0.24, released on 2016.8.29
• Explore. A collection of xap, component and asset that have been built and published. You can add them to your own Studio, reuse or learn from them. I expect there will be better grouping here, such as function type (data wrangling, data mining, data visualization) and language (JavaScript, Python, R).
• Learn. Tutorials, Documentations, Discussion. This is a place you can gain more detailed answers.

Get family with the JavaScript Component and Xap

1. At Studio page, add a JavaScript Component, name it “tutorial”.
2. Open the component, in the “info” tab, you can see the inputs, outputs, layout and dependencies. Their functions are self-explained by the name. And “dependencies” shows a dollar sign and an URL, which means $ is claiming the namespace and the JavaScript library is imported from the URL. In the “edit” tab, you will see more items shown on the left. The most important one is “script”, a place where we will get our hands dirty.
3. Right now we don’t do any changes. Just save and go back to Studio, add a Xap, name it “Xap_0”. Open it by clicking the hammer. Use your curiosity to click everything before we settle down on the “edit” page and “dataflow” tab.
4. On the left bar, click “add component” and drag our beloved “tutorial” component to the blank background. Now the little cute square box appears with the name “tutorial_0”. It has a “doSomthing” input port on the left and a “data” output port on the right. use your curiosity again to explore the box. click the port or double click the box to expand the box.And the button to collapse the box. Mouse hover the box to find the “edit” entrance.
5. Now click the input port and find an arrow button when you mouse hover near the output port. click the arrow to trigger the component. Notice the blue light up on the port, or red flash if something goes wrong. Notice the log icon on the right bottom corner of the page. This is actually the debugging tool for the component. Explore what’s there.
6. click the input port again. Notice that under “doSomething” shows ‘string’, which indicates the data type of the input port. Write some string in the empty underscore. Then click the “…” sign to open the edit page. Replace the script by:

   export default {
       doSomething() {
           let state = this.api.inputState.export();
           let s = state.doSomething;
           this.api.log("LOG",s);
           this.api.output("data", "Hello World");
       }
   };
   

   There are several things to pay attentions here:
   • doSomething() and state.doSomething must match the input port name to correctly trigger the method and capture the data.
   • let is introduced in ECMAScript 6 and is similar to the use of var They have slightly difference in scoping.
   • trigger the component by clicking the arrow at the input. Observe what’s happening in the log message and the output port.

Data type

1. open the scirpt of “tutorial_0” component and replace the line this.api.out() with the following:

   var ent = {
       "dream_car": "Tesla Model 3", 
       "price ": "35000",
         "schedule":"July 2017"
     }; // JSON format/Python dictionary/Java hashmap
   this.api.output ('data',ent);
   

   Trigger the input port and check what you get at the outputs port. After the “data”, you will see “3 attributes”. Click on it and the meat comes out, which is exactly the variable ent that we just defined. Exaptive calls data type as entity. You may feel confused about this fancy name. But it is essentially the same with JSON in JavaScript, or dictionary in Python, or hashmap in Java. For R users, it is similar to a list or 2-column data frame.
2. The value type for input or output that you are allowed to set in Exaptive is: boolean, integer, float, string, tag, null, type, entity, multiset, dynamic, nullable. If you are not sure which one to use, dynamic is the one-size-for-all choice. Try to change the value type of output in the edit page and check whether it still works.
3. Now change the this.api.output ('data',ent); to this.api.output ('data',[ent,ent]); and check what changes at the output. You will see 2 entities/ 3 attributes. Keep in mind that this will be the most frequent datatype we will encounter in the future.

Parsing string

Draw the graph

Note: Sorry I am not able to finish this tutorial, because the position that I have been preparing for a full month is cut right after my final interview. It is a shock to me when I was expecting an offer negotiation and the celebration of my 100th post.

​

2nd Xap I built, AWS S3

Last mile vs first mile

The previous post showed how I built my 1st Xap. I deliberately avoided directly file import. My logical thinking is reconstructed as follows:

1. One line of d3 code d3.csv(file, function) has already integrated 3 steps: open the local or remote file by the string name or URL, parse the file object into JSON format, callback a function to process the data.
2. Although the first 2 steps have the corresponding Exaptive components, building a pure step-3 component could be more difficult than building a one-stop-shopping component.
3. At that time, I hadn’t figure out what exactly is the data format during the flow between components and being sent to the dimple function.
4. To get d3.csv work, I came out a nerdy approach: host the dataset in my own GitHub.

So my 1st Xap is focused on the data visualization, the last mile for a data scientist. “Begin with the End in Mind” is a hard lesson I learned during my past years. After that, I turned my attention to the first mile. I want to be very clear about the data format in each step and I would like to peek into every black box.

Gear up: WebStorm, AWS S3

Then I looked for a Javascript IDE, something like Jupyter notebook in Python. WebStorm caught my attention. It offers 30-day free trial and is free for 2 years for students and teachers, training and open source projects. Even for a standard individual customer, the annual fee of $59 is affordable.

After playing for a while, I realized WebStorm is good for developing the complex function of a simple input, something more pure for javascript coding. But for a web-based application, you always have to deal with file input and script input. If you don’t have a html file, the powerful libraries like jQuery and D3 are sitting on the bench because they are born to manipulate the DOM.

So I return to my previous tool set: start up a local host by python, use atom to write html/javascript codes and check the visual effect on Chrome.

Once I handcraft my html pages, I would like to have a remote host to display them. Years ago Google Drive provided such service for free. Now the hosting business is taken by AWS. I am surprised to see AWS provides up to 17 categories of services such as: compute, storage, database, developer tools, management tools, analytics, AI, Internet of things. S3 (simple static storage) is only a tiny business. By the way, the AWS page looks ugly. I guess because it is like a warehouse shopping center, the target customers care more about the price and stability, rather than a sexy face.

The entrance is: https://console.aws.amazon.com/s3

My endpoint is: http://jychstar.s3-website-us-west-2.amazonaws.com/

Here are my examples:

https://s3-us-west-2.amazonaws.com/jychstar/world_cup_all_in_one.html

https://s3-us-west-2.amazonaws.com/jychstar/aboutme/About_Me.html

https://s3-us-west-2.amazonaws.com/jychstar/fresh_tomatoes.html

Note: For security reasons. the browsers block the Javascript codes. To unblock, in Chrome, there is a shield logo, click on it and “load unsafe scripts”. In firebox, there is “i” sign to fix. In Safari, sorry I don’t know how to solve it.

Papa.parse

Papa parse seems one of the most popular JavaScript libraries that do the parsing job. 4000 stars in Github. Major development around 2014. By the way, this implies it was at the time when the author, Matt Holt, was working at SmartyStreets(a company providing address info) and an undergraduate student at Brigham Young University.

I love the style of its official website: http://papaparse.com/ It uses friendly dialogues to provide case-by-case solution. So the users can quickly pin down what they want, whether it is parsing csv-format string, local file, remote file or even convert JSON back to csv. Actually, it is smart enough that it can find the right delimiter by scanning the first few rows.

var results = Papa.parse(csvString);
console.log(results.meta.delimiter);

However, there is a pitfall when parsing a local file:

Papa.parse(file, {
    complete: function(results) {
        console.log(results);}
});

According to the documentation, file is a File object obtained from the DOM. You can’t just use “data.csv” and hope it can do the magic. Papa will think “data.csv” is only a string and tell you it can’t find a delimiter. In this sense, d3 is much smarter in that d3.csv("data.csv", callback) works for local file.

Anyway, the lower-level of Papa.parse means it has more flexibility to manipulate the data stream. In a github issue discussion , Holt provides some codes that use jQuery AJAX call to pass a file:

$.get("/basic_charts/train.csv", function(text){
    var data = Papa.parse(text);
    console.log(data);
});

However, the problem by this AJAX request is that the whole file is loaded into memory. If the file is too large, the browser gets crash. Alternatively, Papa uses HTML5’s FileReader API to “stream” in the file, if a <input type="file"> element is used in the HTML file. Thanks to Raffael Vogler for providing a pain-free tutorial

<script src="https://cdnjs.cloudflare.com/ajax/libs/PapaParse/4.1.2/papaparse.js"></script>
<script src="https://code.jquery.com/jquery-3.1.1.min.js"></script>
<script>
function handleFileSelect(evt) {
    var file = evt.target.files[0];
    Papa.parse(file, {
        header: true,
        dynamicTyping: true,
        complete: function(results) {
            console.log(JSON.stringify(results, null, 2));}
    });
}
$(document).ready(function(){
    $("#csv-file").change(handleFileSelect);
});
</script>
<!--To get started we need a button to open a file: -->
<input type="file" id="csv-file" name="files"/>

console.log()is only able to print simple data formats such as string, number or array. JSON.stringify is a very powerful debugger tool that you can print the original format of an object. The only thing you can’t print is the function.

debug finding:

1. jQeury change() is triggered by any change made to <input>, \<textara> and \<select> elements.
2. evt is a jQuery object, has 9 top-level keys, one of them is target. Below target is an encoded JQuery file object.
3. results has 3 top-level keys: data, errors, meta. The JSON array is under data key.
4. add indention in stringify parameter for pretty print.

If the file is written in JSON format or JSON array,

var xmlhttp = new XMLHttpRequest();
xmlhttp.onreadystatechange = function() {
    if (this.readyState == 4 && this.status == 200) {
        myObj = JSON.parse(this.responseText); // read raw text
        console.log(JSON.stringify(myObj));
    }
}; // define action
xmlhttp.open("GET", "data.txt", true);
xmlhttp.send();

TitanicXap2

The ready-to-use Xap is here, you can download the csv file from here). Drop the file to see the magic.

The block diagram is:

FileDropTarget:
    CSVParser:
        dimple_barplt
        Titanic_python: pie_plotly

I am going to document my source code.

For the whole Xap, the page DOM is configured in Edit-style-HTML. We can see there are three visible objects.

<div data-node="FileDropTarget_0"></div>
<div data-node="dimple_barplot_0" id="barplot"></div>
<div data-node="pie_plotly_0" id = "pie"></div>

dimple_barplot

Input: data: multiset

spect: dependencies: jQuery, d3, dimple

script: inside export default{data(){},};

let d3 = this.api.imports.d3;
let dimple = this.api.imports.dimple;
let data = this.api.inputState.export().data; // JSON Array
//this.api.log("I am Cool: ", JSON.stringify(data)); // test the raw data
function draw(data) {
  "use strict";
  var svg = dimple.newSvg("#barplot", 600, 400);  // svg inside id barplot with size (600,400)
  svg.append("text").
  attr("x", 300).attr("y", 20).attr("text-anchor", "middle").style("font-size", "20px").style("font-weight", "bold").text("Titanic Survivor");
  var myChart = new dimple.chart(svg, data);
  var x = myChart.addCategoryAxis("x", "Pclass");
  var y = myChart.addPctAxis("y", "Survived");
  var s = myChart.addSeries("Sex", dimple.plot.bar);

  s.afterDraw = function (shape, data) {
    var s = d3.select(shape),
      rect = {
        x: parseFloat(s.attr("x")),
        y: parseFloat(s.attr("y")),
        width: parseFloat(s.attr("width")),
        height: parseFloat(s.attr("height"))
      };
    if (rect.height >= 8) {
      svg.append("text")
        .attr("x", rect.x + rect.width / 2)
        .attr("y", rect.y + rect.height / 2 + 3.5)
        .style("text-anchor", "middle")
        .style("font-size", "10px")
        .style("font-family", "sans-serif")
        .style("opacity", 0.6)
        .style("pointer-events", "none")
        .text(data.yValue);
    }
  }; // end s.afterDraw
  myChart.addLegend(150, 10, 380, 20, "right");
  myChart.draw();
}// end function draw
draw(data); // call function and feed data with JSON Array format

The majority of the code is from dimple github.

Titanic_python

Input: data: dynamic

output: feature_weight: enitity

SPEC. Tricks:

1. sklearn is based on scipy;
2. json can’t be install by pip
3. pandas is 0.19.2, numpy is 1.12, sklearn is 0.18

"dependencies":{
      "apt": [],
    "pip": [
      {"path": "numpy"},
      {"path": "pandas"},
      {"path": "scikit-learn"},
      {"path": "scipy"}
    ],
    "file":[]
},

script

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score,fbeta_score
import numpy as np
import sklearn
import pandas as pd

def data(self):
    data = self.api.inputstate.export()['data'] 
    # self.api.log("I am Cool: ", data) # test data
    df = pd.DataFrame(data)  # convert JSON array to dataframe
    df['Age'] = df['Age'].apply(pd.to_numeric, args=('coerce',)) # convert to number
    train_data = df

    # Data preprocessing
    # use median age to fill missing value
    def get_median_ages(df):
        median_ages = np.zeros((2,3))
        for j in range(0, 3):
            median_ages[0,j] = df[(df['Sex'] == 'female') & \
                                  (df['Pclass'] == j+1)]['Age'].dropna().median()
            median_ages[1,j] = df[(df['Sex'] == 'male') & \
                                  (df['Pclass'] == j+1)]['Age'].dropna().median()
        return median_ages
    def data_clean(df, median_ages):
        df['Gender'] = df['Sex'].map( {'female': 0, 'male': 1} ).astype(int)
        # use median age to fill the missing data
        for i in range(0, 2):
            for j in range(0, 3):
                  df.loc[ (df.Age.isnull()) & (df.Gender == i) & (df.Pclass == j+1),\
                        'Age'] = median_ages[i,j]

        droplist = ['Name','Ticket','Cabin','Embarked','Sex'] # reserve ID for check
        features = df.drop(droplist, axis = 1)
        return features

    median_ages = get_median_ages(train_data)
    train_cleaned = data_clean(train_data,median_ages)
    features = train_cleaned.drop(['PassengerId','Survived'],axis=1)
    labels = train_data ['Survived']

    # machine learning to predict a target feature: Survived
    # use train-test split to generate 2 sets of data
    X_train, X_test,y_train,y_test = train_test_split(features, labels, test_size=0.3, random_state=0)
    clf=DecisionTreeClassifier()
    clf.fit(X_train,y_train)
    pred=clf.predict(X_test)
    self.api.log("test score:",accuracy_score(y_test, pred))  # 0.81, seems good

    feature_weight={}
    for i,key in enumerate(X_train.columns.values):
        feature_weight[key] = clf.feature_importances_[i]
    self.api.output("feature_weight", feature_weight)

Note that in line 11, df = pd.DataFrame(data) convert JSON array to dataframe. This is the most important glue code that connects JavaScript to Python.

pie_plotly

plotly documentation: https://plot.ly/javascript/reference/

Initially I tried dimple for an hour or so, but got a lot of bug. Then I switch to plotly and it was awesome!

input: data: entity

spec:

"dependencies": {
    "file": [
        {
            "type": "js",
            "path": "https://cdn.plot.ly/plotly-latest.min.js",
            "name": "Plotly"
        },
        {
            "type": "js",
            "path": "https://d3js.org/d3.v4.min.js",
            "name": "d3"
        },
        {
            "type": "js",
            "path": "https://cdnjs.cloudflare.com/ajax/libs/numeric/1.2.6/numeric.min.js",
            "name": "numeric"
        }
    ]
},

script:

export default {
    data() {
    let data = this.api.inputState.export().data;
    let d3 = this.api.imports.d3;
    let Plotly = this.api.imports.Plotly; 

    var labels = [];
    var values = [];
    var row;
    for (row in data) {        
        labels.push(String(row));
        values.push(data[row]);
    }

    var d = {}; // dictionary to store labels, values, type
    d.labels = labels;
    d.values = values;
    d.type  = "pie";

    var layout = {
      width: 500,
      height: 400,
      title : "Feature importance",
      titlefont : 20
    };

    var li = [d];
    Plotly.newPlot('pie', li, layout);
    this.api.log(labels[0]);
    }
};

​