Introduction to Jinja Template and Google Chart

Posted on October 29, 2023 in python

Introduction

In this post you will learn

Why we need a template system
Basic syntax of Jinja Template (Python Jinja2)
Generate a simple HTML report with Jinja and Google Chart (Google Chart)

Why we need a template system

While string formatting in Python is pretty straight forward e.g. print(f"Hello {name}"), it could be quite troublesome when we are writing some long messages with many variables, or sometimes we want to use Python function like if or for loop in the paragraph.

A template system in Python, or any programming language, is a tool that allows you to separate the presentation logic (Template) from business logic (Code) in your applications. It provides a way to define and generate content, such as HTML, XML, or plain text, by combining static template files with dynamic data. In this post, we will demostrate how to generate some simple HTML report in the following section.

Basic Syntax of Jinja Template

Render your first jinja template

It looks very similar to string formatting at the first glance. We will first define a template t = Template(...) with variable {{ var_name }} , and then render it with the python variables t.render(var_name="sth").

from jinja2 import Template

t = Template('Hello, {{ name }}!')
print(t.render(name='John Doe'))

# Output:
"Hello, John Doe!"

Basic syntax

The syntax is almost the same as python with a little bit of syntactic sugar, we can use some conditional block via {% %}, and reference to the variable with {{ }}. And another useful thing to note is, we can add a dash - to the operator, so Jinja knows we dont want an additional line break, e.g. {%- -%} (No line break) vs {% %} (With line break).

a) Variables

Syntax: {{ foo }}, {{ foo.bar }}, {{ foo["bar"] }}

You can use a dot (.) to access attributes of a variable in addition to the standard Python __getitem__ “subscript” syntax ([]).

from jinja2 import Template

baz = "a"
foo = {}
foo["bar"] = "b"

t = Template("""
{{ baz }}
{{ foo.bar }}
{{ foo["bar"]}}
""")

print(t.render(
    foo=foo,
    baz=baz,
))

# Output
a
b
b

b) For loop

Syntax: {% for i in some_list %} {{ i }} {% endfor %} with line break
and {%- for i in some_list -%} {{ i }} {% endfor %} without line break

from jinja2 import Template

some_students = ["john", "terry", "ken"]

t = Template("""
{%- for student in students -%}
<li> {{ student }} </li>
{% endfor %}
""")
print(t.render(students=some_students))

# output
<li> john </li>
<li> terry </li>
<li> ken </li>

c) If… elif.. else

Syntax: {% if x = "a" %} {% elif x = "b" %} {% else %} {% endif %}

from jinja2 import Template

x = 40

# Additional spacing is added for readability
t = Template("""
{% if x >= 0 and x < 30 %}
x is larger than 0

{% elif x >= 30 %}
x is larger than 30

{% else %}
x is smaller than 30

{% endif %}
""")
print(t.render(x=x))

# output
x is larger than 30

d) Comments

Syntax: {# #} (with new line) or {#- -#} (without new line)

from jinja2 import Template

t = Template("""
{#- This is a comment. And it is not displayed in the output. -#}
Hello, {{ name }}!
""")
print(t.render(name='John Doe'))

# Output
Hello, John Doe!

e) Read template from a file

For better file management, we usually put the files into a ./templates folder. It helps to separate the presentation layer (template) and logical layer (code). The file structure would be something like this.

.
├── main.py
└── templates
    └── simple_report.html

First we put the template file under ~./templates/simple_report.html

{#- <!-- templates/simple_report.html --> -#}
<!DOCTYPE html>
<html>
  <body>
    <h1>Hi {{ name }}!</h1>
  </body>
</html>

Then we can read the templates with open("some_file.html", "r") as f: ... Template(f.read()) in main.py

# Easy version
# May mess up Jinja Template Hierarchy tho. In our simple case, it wont matter much
from jinja2 import Template

name = "John"

with open("templates/simple_report.html", "r") as f:
    template = Template(f.read())

rendered = template.render(name=name)
print(rendered)

A more proper version from the API documentation with Environment and PackageLoader/FileSystemLoader. For our simple case, it looks like an over kill.

Ref: API Documentation and StackOverflow

# Proper version
from jinja2 import Environment, FileSystemLoader, select_autoescape

env = Environment(loader=FileSystemLoader(searchpath="./"))

name = "John"
template = env.get_template("templates/simple_report.html")

rendered = template.render(name=name)
print(rendered)

Both version will render the same result.

<!DOCTYPE html>
<html>
  <body>
    <h1>Hi John!</h1>
  </body>
</html>

Simple Reporting Templates with Google Chart

After the long syntax introduction, we will jump straight to the use cases.

Sometimes when the buisness users are asking for some advance/deep-dive adhoc reports, there are not much report choices for a quick study, you can use

Excel file
Commercial products (e.g. Tableau, PowerBI, etc..)
Self host a Python server (e.g. matplotlib, seaborn, etc..)
A simple HTML file generated with Template and some Javascript Library (Below example)

Each of the option has it’s pros and cons, a static file (like HTML) provides you an other options, if other commercial tools or hosting a python server is not available.

Google Chart Library

Google chart tools are powerful, simple to use, and free. And usually these kinds of Javascript chart library provides a richer (and customizable) analytic gallery for you to use, which sometimes it is quite difficult to do it in excel, like Sankey Diagram, Interval Plots and Tree Map.

Here I will use a simple bar chart as an example. And you will see how we can create a simple HTML file with bar chart (or other graphs) easily if you know how to use Jinja Template.

This is the outline of the steps to create a standalone HTML file with Google Chart

Copy the minimal example of Stacked bar chart (Google Charts)
Replace the data part with Jinja Template variable
Try it with dummy data
Replace it with real data. Here we will use the Daily Passenger Traffic for different Control Points from HK Government Open Data (data.gov.hk)
Render the Template, and export as HTML file

a) Stacked Bar Chart

Let’s copy the minimal example of stack bar chart. I rename some variables name from the original example here (e.g. materialChart to chart, etc..)

The code is pretty easy to read. We import some chart library with <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script> and initialize the google chart object and callback function with with google.charts.setOnLoadCallback(drawChart);. It’s quite straight forward even if you do not have much experience with Javascript.

<html>
  <head>
    <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
    <script type="text/javascript">

     google.charts.load('current', {packages: ['corechart', 'bar']});
     google.charts.setOnLoadCallback(drawChart);

     function drawChart() {
       var data = google.visualization.arrayToDataTable([
         ['City', '2010 Population', '2000 Population'],
         ['New York City, NY', 8175000, 8008000],
         ['Los Angeles, CA', 3792000, 3694000],
         ['Chicago, IL', 2695000, 2896000],
         ['Houston, TX', 2099000, 1953000],
         ['Philadelphia, PA', 1526000, 1517000]
       ]);

       var options = {
         chart: {
           title: 'Population of Largest U.S. Cities'
         },
         hAxis: {
           title: 'Total Population',
           minValue: 0,
         },
         vAxis: {
           title: 'City'
         },
         bars: 'horizontal'
       };
       var chart = new google.charts.Bar(document.getElementById('chart_div'));
       chart.draw(data, options);
     }
    </script>
  </head>

  <body>
    <div id="chart_div"></div>
  </body>
</html>

b) Replace with Jinja Template variables

Then we can replace some of the values with Template variable like {{ data }}, {{ title }}, {{ h_axis }}, etc..

from jinja2 import Template

t = Template(
"""
<html>
  <head>
    <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
    <script type="text/javascript">

     google.charts.load('current', {packages: ['corechart', 'bar']});
     google.charts.setOnLoadCallback(drawChart);

     function drawChart() {
       var data = google.visualization.arrayToDataTable(
       {{ data }}
       );

       var options = {
         chart: {
           title: '{{ title }}'
         },
         hAxis: {
           title: '{{ h_axis }}',
           minValue: 0,
         },
         vAxis: {
           title: '{{ v_axis }}'
         },
         bars: 'horizontal'
       };
       var chart = new google.charts.Bar(document.getElementById('chart_div'));
       chart.draw(data, options);
     }
    </script>
  </head>

  <body>
    <div id="chart_div"></div>
  </body>
</html>
"""
)

c) Test with Dummy Data

If we go back and study the embeded data in the json, we will see the data we need is a list of list, with 3 columns (One categorical and two numerics). So we we create the dataframe accordingly, with ["Control Point", "Arrival", "Departure"] as our column for our traffic data.

# The embeded data in the original json. A list of list with each record as an element.
[
  ['City', '2010 Population', '2000 Population'],
  ['New York City, NY', 8175000, 8008000],
  ['Los Angeles, CA', 3792000, 3694000],
  ['Chicago, IL', 2695000, 2896000],
  ['Houston, TX', 2099000, 1953000],
  ['Philadelphia, PA', 1526000, 1517000]
]

Let’s create a dummy dataframe, and convert it to list of list object.

import pandas as pd

df = pd.DataFrame({
    "Control Point": ["Airport", "Lok Ma Chau", "Lo Wu"],
    "Arrival": [154, 120, 40],
    "Departure": [21, 40, 32],
})
print(df.head())

# Convert dataframe to List of list, and insert to column header as the first element.
data = df.values.tolist()
data.insert(0, df.columns.tolist())
print(data)

# df
  Control Point  Arrival  Departure
0       Airport      154         21
1   Lok Ma Chau      120         40
2         Lo Wu       40         32

# data
[
  ['Control Point', 'Arrival', 'Departure'],
  ['Airport', 154, 21],
  ['Lok Ma Chau', 120, 40],
  ['Lo Wu', 40, 32]
]

d) Replace with real data

Here we will get the 2022 passenger traffic data from HK Government Open Data (data.gov.hk). And aggregate the data from daily records to yearly records. For the df operation, you can check out my previous post on Common Pandas Functions.

import pandas as pd

# Get the 2022 arrival/departure data
url = "https://www.immd.gov.hk/opendata/eng/transport/immigration_clearance/statistics_on_daily_passenger_traffic.csv"
df = pd.read_csv(url)
df = df[["Date", "Control Point", "Arrival / Departure", "Total"]]
df['Date'] = pd.to_datetime(df['Date'], format="%d-%m-%Y")
df_subset = df[(df['Date'] >= '2022-01-01')
               & (df['Date'] <= '2022-12-31')].reset_index(drop=True)  # 2022 data only, for simplicity
print(df_subset)

# Group by Control Point. We can safely ignore date, as we are only using 2022 data
df_agg = df_subset.groupby(["Control Point", "Arrival / Departure"]).agg(Total=("Total", "sum"),).reset_index()
df_agg = df_agg[df_agg.Total > 0].reset_index(drop=True)
print(df_agg)

# Long to wide. pivot_table is used here, just for the fill_value argument
df_wide = df_agg.pivot_table(
    index=["Control Point"],
    columns="Arrival / Departure",
    values=["Total"],
    aggfunc="sum",
    fill_value=0,
)

df_wide_flattened = df_wide.copy()
df_wide_flattened.columns = ["_".join(x) for x in df_wide_flattened.columns.to_flat_index()]
df_wide_flattened.reset_index(inplace=True)
print(df_wide_flattened)

data = df_wide_flattened.values.tolist()
data.insert(0, df_wide_flattened.columns.tolist())  # Insert back the column header
print(data)

# df_subset (2022 Data)
            Date                   Control Point Arrival / Departure  Total
0     2022-01-01                         Airport             Arrival    628
1     2022-01-01                         Airport           Departure    778
2     2022-01-01  Express Rail Link West Kowloon             Arrival      0
3     2022-01-01  Express Rail Link West Kowloon           Departure      0
...

# df_agg (Aggregate to Year level)
                    Control Point Arrival / Departure    Total
0                         Airport             Arrival  1957891
1                         Airport           Departure  2175626
8                  Heung Yuen Wai             Arrival      301
10  Hong Kong-Zhuhai-Macao Bridge             Arrival    77075
...

# df_wide_flattened (Long to wide table)
                   Control Point  Total_Arrival  Total_Departure
0                        Airport        1957891          2175626
1                 Heung Yuen Wai            301                0
2  Hong Kong-Zhuhai-Macao Bridge          77075           114504
3        Kai Tak Cruise Terminal           8233             3610
4                   Shenzhen Bay         485248           439669

# data (Final List of List object for input of the graph)
[['Control Point', 'Total_Arrival', 'Total_Departure'], ['Airport', 1957891, 2175626], ['China Ferry Terminal', 0, 0], ..]

e) Render with Jinja Template, and export to html file

Finally, we can render the variables with Jinja, and export to the HTML file.

from jinja2 import Template
import pandas as pd

# Get the 2022 arrival/departure data, hopefully the URL will not break
url = "https://www.immd.gov.hk/opendata/eng/transport/immigration_clearance/statistics_on_daily_passenger_traffic.csv"
df = pd.read_csv(url)
df = df[["Date", "Control Point", "Arrival / Departure", "Total"]]
df['Date'] = pd.to_datetime(df['Date'], format="%d-%m-%Y")
df_subset = df[(df['Date'] >= '2022-01-01')
               & (df['Date'] <= '2022-12-31')].reset_index(drop=True)  # 2022 data only, for simplicity

# Group by Control Point. We can safely ignore date, as we are only using 2022 data
df_agg = df_subset.groupby(["Control Point", "Arrival / Departure"]).agg(Total=("Total", "sum"),).reset_index()
df_agg = df_agg[df_agg.Total > 0].reset_index(drop=True)

# Long to wide
df_wide = df_agg.pivot_table(
    index=["Control Point"],
    columns="Arrival / Departure",
    values=["Total"],
    aggfunc="sum",
    fill_value=0,
)

df_wide_flattened = df_wide.copy()
df_wide_flattened.columns = ["_".join(x) for x in df_wide_flattened.columns.to_flat_index()]
df_wide_flattened.reset_index(inplace=True)

# List of List
data = df_wide_flattened.values.tolist()
data.insert(0, df_wide_flattened.columns.tolist())  # Insert back the column header

# Create template
t = Template("""
<html>
  <head>
    <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
    <script type="text/javascript">

     google.charts.load('current', {packages: ['corechart', 'bar']});
     google.charts.setOnLoadCallback(drawChart);

     function drawChart() {
       var data = google.visualization.arrayToDataTable(
       {{ data }}
       );

       var options = {
         chart: {
           title: '{{ title }}'
         },
         hAxis: {
           title: '{{ h_axis }}',
           minValue: 0,
         },
         vAxis: {
           title: '{{ v_axis }}'
         },
         bars: 'horizontal'
       };
       var chart = new google.charts.Bar(document.getElementById('chart_div'));
       chart.draw(data, options);
     }
    </script>
  </head>

  <body>
    <div id="chart_div"></div>
  </body>
</html>
""")

# Render template, and save to a html file
output = t.render(
    data=data,
    title="Inbound and Outbound Passenger at different Control Points (Year 2022)",
    h_axis="Passengers",
    v_axis="Control Point",
)
with open('output.html', 'w') as f:
    f.write(output)

Output - output.html

Final Thoughts

There is a lot of other Javascript chart packages (e.g. Vega Chart, HighCharts, etc..), you can easily create some pretty charts with the techniques here with Jinja Template. As it is in HTML format, you can easily include some Insight session as well for some quick comments for your report.

Remarks: Just be careful for not leaking any sensitive data, as a static report is difficult to do access control or audit logging.

Happy Coding!

Recommended Readings

18 Useful Pandas Functions for Data Science

Here is my list of useful pandas functions for your day to day tasks.

Getting HKEX data with Quandl in Python

Getting HKEX data with Quandl in Python. Historical daily HKEX data using API. Stock exchange in Yahoo Finance Hong Kong.

Reference

Photo by 1981 Digital on Unsplash