jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data

To run this demo, your system needs at least:

This demo uses stackable operators to deploy a Spark Connect server and an HDFS cluster. The intention is to demonstrate how clients, in this case a JupyterLab notebook, can interact with SDP components. The notebook creates a SparkSession, that delegates the data analysis and model training to a Spark Connect service thus offloading resources into the Kubernetes cluster. The notebook can thus be used as a sandbox for writing, testing and benchmarking Spark jobs before they are moved into production.

This demo will:

The Stackable Operator for Apache HDFS will spin up an HDFS cluster to store the taxi dataset in Apache Parquet format. This dataset will be read and processed via PySpark.

Before trying out the notebook example in Jupyter, check if the taxi data was loaded to HDFS successfully:

There should be one parquet file containing taxi trip data from September 2020.

Have a look at the available Pods before logging in:

In order to reach the JupyterLab web interface, create a port-forward:

The jupyterlab service is created along side the the JupyterLab deployment.

Now access the JupyterHub web interface via http://localhost:8080

You should see the JupyterLab login page.

Log in with token adminadmin. You should arrive at your workspace:

Now you can double-click on the notebook folder on the left, open and run the contained file. Click on the double arrow (⏩️) to execute the Python scripts (click on the image below to go to the notebook file).

You can also inspect the hdfs folder where the core-site.xml and hdfs-site.xml from the discovery ConfigMap of the HDFS cluster are located.

The Python notebook uses libraries such as pandas and scikit-learn to analyze the data. In addition, since the model training is delegated to a Spark Connect server, some of these dependencies, most notably scikit-learn, must also be made available on the Spark Connect pods. For convenience, a custom image is used in this demo that bundles all the required libraries for both the notebook and the Spark Connect server. The source of the image is available here.

In practice, clients of Spark Connect do not need a full-blown Spark installation available locally, but only the libraries that are used in the notebook.

The job uses an implementation of the Isolation Forest algorithm provided by the scikit-learn library: the model is trained and then invoked by a user-defined function (see this article for how to call the sklearn library with a pyspark UDF), all of which is run using the Spark Connect executors. This type of model attempts to isolate each data point by continually partitioning the data. Data closely packed together will require more partitions to separate data points. In contrast, any outliers will require less: the number of partitions needed for a particular data point is thus inversely proportional to the anomaly "score".

The notebook shows how to plot the outliers against a particular metric (e.g. "number of rides"):

However, this is mainly for convenience - the anomaly score is derived from the entire feature space, i.e., it considers all dimensions (or features/columns) when scoring data, meaning that not only are the results challenging to visualize (how can multidimensional data be represented in only 3-D dimensional space?), but that a root cause analysis has to be a separate process. It would be tempting to look at just one metric and assume causal effects, but the model "sees" all features as a set of numerical values and derives patterns accordingly.

We can tackle the first of these issues by collapsing - or projecting - our data into a manageable number of dimensions that can be plotted. Once the script has finished successfully, plots should be displayed on the bottom that show the same data in 2D and 3D representation. The 3D plot should look like this:

The model has detected outliers even though that would not have been immediately apparent from the time-series representation alone.