Python is awesome because you can do anything. In this case, I’m going to show you how to configure a Jupyter notebook with integrated Plotly support to draw interactive graphs. This is very useful if you have to analyse something.

Introduction

In case you don’t know it, the Jupyter project exist to develop open-source software, open-standards, and services for interactive computing across dozens of programming languages. All started with Python and R, but nowadays, it is compatible with almost any popular language.

I’m going to focus in how to create a Jupyter notebook for Python using a Docker image. This will allow you to replicate the environment wherever you want with a single command and keep all the dependencies in a single place.

One of the key things is that I’m going to explain how to integrate the Plotly library into the notebook so we will be able to create interactive plots that can be exported with the notebook itself. This create an amazing tool to share reports within your organization.

Docker environment

Finding the images is easy. If you google it, you will find the repository jupyter/docker-stacks that contains the images that you need. but, what do you need? The Jupyter environment is growing fast and sometimes is hard to know what are we looking for. The best place to start, as always, is the documentation page. Skipping the examples that shows you how to create a notebook in a single command, I’m going to enter directly in the section “Selecting an image” to understand the difference and select what we need. Go to the link to read about the images and then come back here. Keep in mind that I’m looking for a Python environment with at least Pandas and Plotly installed.

The image “jupyter/base-notebook” could be a good options, but there are still useful missing dependencies provided by “jupyter/minimal-notebook”. “jupyter/r-notebook” is discarded, we want Python. “jupyter/scipy-notebook” contains everything we need except Plotly. It even contain more things that we need, like seaborn. The following images diverge further more from our needs so I’m going to use the jupyter/minimal-notebook and install the dependencies myself. Let’s create a Docker container.

The Base Image

The documentation warns that the images are regularly updated with backward incompatible changes. For this reason, it is recommendable to use a constant docker tag instead of the $LATEST reference. You can go directly to docker hub and find the current latest tag to start a new project. In this moment, the tag is “jupyter/minimal-notebook:ad3574d3c5c7”. Let’s download it and create a minimal notebook. The following command will do it.

1

docker run -p 8888:8888 jupyter/minimal-notebook:ad3574d3c5c7

When it finishes, you will have the notebook running in http://localhost:8888. Check it out. The notebook requires you to give the token that has been printed in the terminal. Just copy and paste it.

Now you can create a new Python notebook by clicking on the top left button “new” and selecting “Python 3”. This will open a new windows and you will be ready to go. Try to type a few Python commands and press Shift+Enter. This will execute the command and move to the next cell.

1

print("Hello world!")

This is cool, but we need a lot of dependencies. If you try to run import pandas, it will complain. To install the dependencies, we need a Dockerfile and a requirements.txt.

The Docker file

It is time to create a folder somewhere in your system to put everything on it. I would recommend you to put it under VSC with git. The first thing you should create is a Dockerfile and a requirements.txt with the following content.

1
2
3
4
5

FROM jupyter/minimal-notebook:ad3574d3c5c7

WORKDIR /tml
COPY requirements.txt .
RUN pip install -r requirements.txt

1
2

pandas
plotly

Now we can give it a try by building and running the container.

1
2
3

# from the folder containing the files
docker build -t jupyter .
docker run --rm -it -p 8888:8888 jupyter 

This will print in the terminal a series of messages. You should look for the one displaying the localhost URL and the token. If you click the link, It will open the browser with your new jupyter notebook environment.

[I 09:32:25.749 NotebookApp] The Jupyter Notebook is running at:
[I 09:32:25.749 NotebookApp] http://75fcb2e2b029:8888/?token=da7ef85e9afb1568794b3c222f633c144ba9c010f9de26b2
[I 09:32:25.749 NotebookApp]  or http://127.0.0.1:8888/?token=da7ef85e9afb1568794b3c222f633c144ba9c010f9de26b2

So it is done! We can star playing with it.

Enable JupyterLab (Optional)

Looks like the world is always moving and now we have the JupyterLab web interface instead of the classic Jupyter Notebook. We can easily enable it by adding an environment variable at the end of the Dockerfile. It is still a little bit unstable but I think that It will be quickly improved.

Also, I’ve added some essential extensions for me directly in the dockerfile.

1
2
3
4
5
6

ENV JUPYTER_ENABLE_LAB=yes

RUN jupyter labextension install jupyterlab_vim && \
    jupyter labextension install @jupyterlab/dataregistry-extension && \
    pip install --upgrade jupyterlab-git && \
    jupyter lab build

From my point of view, this is a must just for the vim extension.

Setting up Plotly

By default, Plotly should work without any extra setup. The problem is that in some occasions you have to change the render configuration. This is really well explained in the Plotly Renderers documentation. You can use the following snippet to check your default render and to modify it

1
2
3
4
5
6
7
8

# [0]
# print
import plotly.io as pio
pio.renderers

# [1]
# modify
pio.renderers.default = "iframe"

It’s time to run a sample code to see if we can draw a plot. Just copy and paste this into a cell.

1
2
3
4
5
6

import plotly.graph_objects as go
fig = go.Figure(
    data=[go.Bar(y=[2, 1, 3])],
    layout_title_text="A Figure Displaying Itself"
)
fig.show()

You should see something like this, but with a different style.

Dict syntax vs graph_objects

In the previous example, the figure is build using the graph_objects. This is not a bad idea, but for simple cases, I prefer to create the dict by my self. This is basically the same idea. Let’s compare.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

import plotly.io as pio
fig = {
    "data": [
        {
            "type": "bar",
            "y": [ 2, 1, 3 ]
        }
    ],
    "layout": {
        "title": "A Figure Displaying Itself"
    }
}
pio.show(fig)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import plotly.graph_objects as go
fig = go.Figure(
    data=[
      go.Bar(
        y=[2, 1, 3]
      )
    ],
    layout_title_text="A Figure Displaying Itself"
)
fig.show()

If you want to get the json representation of the graph_objects, there is a method to_json that is what you expect. There is also an option to convert the dict to a graph_objects using go.Figure and passing the dictionary as parameter.

Working with Pandas

The next big question is. How do we draw pandas dataframes? Thankfully, we have Python List Comprehensions that make things much easier. First let’s fetch some data. I’ve chossen data from Airbnb Barcelona from here. Loading the data is straight forward thanks to pandas. The second instructions is to parse the price values into float values to allow calculations.

1
2
3

import pandas as pd
df = pd.read_csv("http://data.insideairbnb.com/spain/catalonia/barcelona/2019-09-17/data/listings.csv.gz")
df["price_value"] = df.price.replace('[\$,]', '', regex=True).astype(float)

Okey, now let’s draw the mean price of a room based on the number of room. I like to wrap my plots with a function, but this is just personal preference.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

def beds_vs_meanprice(df: pd.DataFrame):
    data = df.groupby("beds",as_index=False)["price_value"].mean()
    fig = {
        "data":[
            {
                "type": "bar",
                "x":data["beds"],
                "y":data["price_value"]
            }
        ]
    }
    pio.show(fig)
beds_vs_meanprice(df)

This produces an ugly plot, but let’s focus on just the data for now.

This is really easy to follow, you just have to put the columns that you want to draw and it works! Provably you will find yourself wanting to add categories, so let’s do it. This is where list comprehension is essential. Let’s include a category by the cancellation policy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

def beds_vs_meanprice(df: pd.DataFrame):
    data = df.groupby(["cancellation_policy","beds"],as_index=False)["price_value"].mean()
    fig = {
        "data":[
            {
                "type": "bar",
                "x":data[data["cancellation_policy"]==policy]["beds"],
                "y":data[data["cancellation_policy"]==policy]["price_value"],
                "name":policy
            } for policy in data["cancellation_policy"].unique()
        ]
    }
    pio.show(fig)
beds_vs_meanprice(df)

This may look complicated, but the idea behind is really simple. We get the unique policy values and create a trace for each one. To select the appropriate data, we filter the dataframe by the given policy. To identify it, we assign the trace name the value of the policy.

with this technique, you can draw almost anything in a dataframe using plotly. If you are familiar with working with big datasets, you may notice that working in this way is not efficient. The good new are that you will hit plotly limitations before you have to think a better way of generating the plot.

Export figure as json

To draw the figures in the blog, I’ve exported the figure in json format and used the plot shortcode. This is cool because allows me to generate complex structures like the category groupby from the previous example and display it here in a dynamic plot with just a copy/paste operation.

fist thing is to import the json package and modify the plot code like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

def beds_vs_meanprice(df: pd.DataFrame):
    data = df.groupby(["cancellation_policy","beds"],as_index=False)["price_value"].mean()
    fig = {
        "data":[
            {
                "type": "bar",
                "x":data[data["cancellation_policy"]==policy]["beds"].to_list(),
                "y":data[data["cancellation_policy"]==policy]["price_value"].to_list(),
                "name":policy
            } for policy in data["cancellation_policy"].unique()
        ]
    }
    print(json.dumps(fig))
    pio.show(fig)
beds_vs_meanprice(df)

the to_list() is important because if not, the json generated contains the dataframe data. Then the json.dumps just works. Here is the output.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

{
    "data": [
        {
            "type": "bar",
            "x": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16 ],
            "y": [ 58.049180327868854, 137.5415173237754, 138.96512935883015, 393.92602040816325, 293.5128205128205, 357.9247311827957, 210.82432432432432, 155.11764705882354, 1966.3333333333333, 358.42857142857144, 1050.3846153846155, 4160, 454.75, 146, 805
            ],
            "name": "flexible"
        },
        {
            "type": "bar",
            "x": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 26, 28 ],
            "y": [ 89.23809523809524, 65.87909209951987, 116.96257796257797, 187.15055467511885, 208.99554565701558, 189.24479166666666, 258.75, 387.48936170212767, 256.4347826086956, 431.75, 258, 30, 107.5, 899, 20, 25, 300, 740, 15 ],
            "name": "moderate"
        },
        {
            "type": "bar",
            "x": [ 1 ],
            "y": [ 42.5 ],
            "name": "strict"
        },
        {
            "type": "bar",
            "x": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 23, 24, 25, 30 ],
            "y": [ 183.8181818181818, 64.93736903376019, 99.97269624573379, 138.1967654986523, 161.53831231813774, 160.23861566484518, 210.30085959885386, 214.05797101449275, 257.85714285714283, 385.0833333333333, 328.4375, 383.5, 361.2857142857143, 542.5, 342.4, 391.42857142857144, 484.44444444444446, 580, 628, 453, 500, 550, 545.5, 483 ],
            "name": "strict_14_with_grace_period"
        },
        {
            "type": "bar",
            "x": [ 1, 2, 3, 4, 5, 6, 7, 8 ],
            "y": [ 625.6923076923077, 222.1764705882353, 615.170731707317, 386.4418604651163, 234.85714285714286, 236.15384615384616, 374.75, 1092.25 ],
            "name": "super_strict_30"
        },
        {
            "type": "bar",
            "x": [ 1, 2, 3, 4 ],
            "y": [ 595.6666666666666, 200, 361.6666666666667, 700.25 ],
            "name": "super_strict_60"
        }
    ]
}

using this json body in the plot shortcode, generates the graph above.

This is not the only way of generating json. The package plotly.io has an option pio.to_json that is much more robust. This function handles the dataframes without the to_list option and also prints style configuration. The only drawback is that the output is much bigger. This is the plot generated from that json.

docker-compose recipe

And to finish this howto, I’m going to leave a repository with a docker-compose configured to create this environment. This way, you can easily create the environment wherever you want. It also includes a volume under the work folder to ensure that your notebooks survive when you stop the container.

Just keep in mind that this is a basic template. You will need to modify it to include credentials, other libraries, more extenssions… whatever you want.

check it out here

Now just run this, and you are ready to code!

1

docker-compose up --build lab

Conclussion

If you are planning to understand some data, you should start with an environment like this. With Pandas is really easy to transform data. With plotly you can create amazing interactive plots. And if you need anything else, remember that you are using Python and everything is already implemented in Python.

Thank you for reading and feel free to leave a comment bellow, I will be really happy to hear from you.

References

• Jupyter Docker Stacks
• Dockerfile reference
• Creating and Updating Figures
• Plotly renderers

extra

• JupyterLab Interactive