Super-powered notebooks with Python and Quarto

thomasmock.quarto.pub/python-austin

What is Quarto?

https://quarto.org

Quarto is an open-source scientific and technical publishing system that builds on standard markdown with features essential for scientific communication.

• Computations: Python, R, Julia, Observable JS
• Markdown: Pandoc flavored markdown with many enhancements
• Output: Documents, presentations, websites, books, blogs

Literate programming system in the tradition of Org-Mode, Weave.jl, R Markdown, iPyPublish, Jupyter Book, etc.

Origins

• Open Source project sponsored by Posit, PBC (formerly known as RStudio, PBC)
• 10+ years of experience with R Markdown, a similar system that was R-specific, convinced us that the core ideas were sound
• The number of languages and runtimes used for scientific discourse is broad
• Quarto is a ground-up re-imagining of R Markdown that is fundamentally multi-language and multi-engine
• Quarto gets inspiration from both R Markdown and Jupyter, and provides a plain-text option or the use of native Jupyter notebooks

Goal: Computation document

• Documents that include source code for their production
• Notebook AND plain-text flavors
• Programmatic automation and reproducibility

Goal: Scientific Markdown

Goal: Single Source Publishing

Simple Example

---
title: "matplotlib demo"
format:
  html:
    code-fold: true
jupyter: python3
---

For a demonstration of a line plot on a polar 
axis, see @fig-polar.

```{python}
#| label: fig-polar
#| fig-cap: "A line plot on a polar axis"

import numpy as np
import matplotlib.pyplot as plt

r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(
  subplot_kw = {'projection': 'polar'} 
)
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()
```

Simple Example, multi-format

Can be rendered to dozens of output formats with Quarto (via Pandoc):

$ quarto render hello.qmd --to html
$ quarto render hello.qmd --to pdf
$ quarto render hello.qmd --to docx
$ quarto render hello.qmd --to epub
$ quarto render hello.qmd --to pptx
$ quarto render hello.qmd --to revealjs

Feature	Quarto
Basic Formats	html pdf docx
Beamer	beamer
PowerPoint	pptx
HTML Slides	revealjs
Advanced Layout	Quarto Article Layout

Feature	Quarto
Cross References	Quarto Crossrefs
Websites & Blogs	Quarto Websites Quarto Blogs
Books	Quarto Books
Interactivity	Shiny for Python/R Observable.js Jupyter Widgets
Dashboards	Quarto Dashboards
Journal Articles	Out and more coming!

So what is Quarto?

Quarto® is an open-source scientific and technical publishing system built on Pandoc.

• quarto is a language agnostic command line interface (CLI)

thomasmock$ quarto --help

Usage:   quarto
Version 1.4.553

Commands:
  render  [input] [args...] - Render input file(s) to various document types.            
  preview [file] [args...]  - Render and preview a document or website project.          
  publish [provider] [path] - Publish a document or project.

Basic Workflow

Rendering (execute and write to disk):

# plain text qmd
$ quarto render python.qmd
$ quarto render python.qmd --to pdf

# ipynb notebook
$ quarto render python.ipynb
$ quarto render python.ipynb --execute

Preview (execute, write to disk, and maintain a LIVE preview of content):

# plain text qmd
$ quarto preview python.qmd
$ quarto preview python.qmd --to pdf

# ipynb notebook
$ quarto preview python.ipynb
$ quarto preview python.ipynb --execute

IPython

For execution of R, Quarto uses knitr as the engine, but for Python Quarto natively executes Python with Jupyter kernels such as IPython.

• The indicated or default Python Jupyter kernel is bound automatically when {python} executable cells are present. You can set a specific kernel via the YAML:

---
title: "My doc"
date: today
jupyter: python3
---

• IPython executes Python code and transforms it to plain text, graphics, markdown, HTML, etc.

• For interactive sessions, Quarto keeps the Jupyter Kernel resident as a daemon to mitigate startup times.

A .qmd is a plain text file

Metadata (YAML)

format: html
jupyter: python3

format: html
engine: knitr

Code

```{python}
import polars as pl
(mtcars
  .groupby("cyl")
  .agg([(pl.col("mpg").mean())]))
```

```{r}
library(dplyr)
mtcars |> 
  group_by(cyl) |> 
  summarize(mean(mpg))
```

Text

# Heading 1
This is a sentence with some **bold text**, some *italic text* and an 
![image](image.png){fig-alt="Alt text for this image"}.

But Quarto doesn’t have to be plain-text

Rendering pipeline

Plain text workflow (.qmd uses Jupyter kernel to execute cells):

Notebook workflow (defaults to using existing stored computation):

What to do with my existing .ipynb?

You can keep using them! You get to choose whether to use the stored computation OR re-execute the document from top to bottom.

# --execute flag is optional - forces re-execution
quarto render my-favorite.ipynb --to html --execute

Quarto can help convert back and forth between plain text .qmd and .ipynb - kind of like jupytext but specific to Quarto:

quarto convert --help

Usage:   quarto convert <input>
Description:
    Convert documents to alternate representations.

Convert notebook to markdown:                quarto convert doc.ipynb                
Convert markdown to notebook:                quarto convert doc.qmd                  
Convert notebook to markdown, write to file: quarto convert doc.ipynb --output doc.qmd

Stored/frozen computation and reproducibility

1. Jupyter natively approaches this as storing the source code, output file, and cache the resulting computation in a single document (.ipynb which is JSON)

2. Jupyter Cache provides transient caching of cell outputs for a doc (if any cells in doc change, then all of the cells will be re-executed)

3. Quarto’s Freeze feature uses a multi-file approach:

• Source code input (plain text .qmd and/or .ipynb)
• Complete output file (some format like .html or .pdf)
• Frozen computation stored separately by directory and file as .json, allows for permanately saving and re-use of computational outputs across entire project.

Comfort of your own workspace

Auto-completion in RStudio + VSCode

Both RStudio and VSCode with the Quarto extension have rich auto-completion

YAML

Chunk option

Quarto Extensions and Visual/Live Editor

Quarto, unified document layout

quarto render boston-terrier.qmd --to html

quarto render boston-terrier.qmd --to pdf

HTML

PDF

Quarto, unified syntax across markdown and code

Add two images on disk to a two column layout.

::: {layout-ncol=2}
![Surus](surus.png)

![Hanno](hanno.png)
:::

Two plots from code, layout in two columns.

```{python}
#| layout-ncol: 2
#| fig-cap: ["Scatter", "Boxplot"]

from plotnine import ggplot, geom_point, geom_boxplot, aes, stat_smooth, facet_wrap, theme
from plotnine.data import mtcars

# plot 1 in column 1
plot1 = (ggplot(mtcars, aes('wt', 'mpg', color='factor(gear)'))
   + geom_point() + stat_smooth(method='lm')
   + facet_wrap('~gear')).draw(show=True)

# plot 2 in column 2
plot2 = (ggplot(mtcars, aes('cyl', 'mpg', color='factor(cyl)'))
+ geom_boxplot()).draw(show=True)
```

Scatter

Boxplot

Quarto Manuscripts

Live example at Quarto Manuscripts

Article Content

Navigation

Quarto Manuscripts: Notebooks

Notebook View

Quarto dashboards

Dashboard

Quarto dashboards

--- 
title: "Development Indicators by Continent"
author: "Gapminder Analytics Group"
format: dashboard
--- 

```{python}
import plotly.express as px
df = px.data.gapminder()
```

## Row {height=60%}

```{python}
#| title: GDP and Life Expectancy 
px.scatter(  
  df, x="gdpPercap", y="lifeExp", 
  animation_frame="year", animation_group="country", 
  size="pop", color="continent", hover_name="country",
  facet_col="continent", log_x=True, size_max=45, 
  range_x=[100,100000], range_y=[25,90] 
)  
```

## Row {height=40%}

```{python}
#| title: Population
px.area(
  df, x="year", y="pop", 
  color="continent", line_group="country"
)
```

```{python}
#| title: Life Expectancy
px.line(
  df, x="year", y="lifeExp", 
  color="continent", line_group="country"
)
```

1

The document options define the title and author for the navigation bar as well as specifying the use of the dashboard format.

2

Rows and columns are defined using headings. In this example we define two rows and specify their relative sizes using the height option.

3

Computational cells become cards that live within rows or columns. Cards can have an optional title (which here we specify using the title option).

4

The second row includes two computational cells, which are automatically split into two side by side cards.

Interactivity, Jupyter Widgets

import plotly.express as px
import plotly.io as pio
df = px.data.iris()
fig = px.scatter(df, x="sepal_width", y="sepal_length", 
                 color="species", 
                 marginal_y="violin", marginal_x="box", 
                 trendline="ols", template="simple_white")
fig.show()

Interactivity, Observable

Quarto also includes native support for Observable JS, a set of enhancements to vanilla JavaScript created by Mike Bostock (also the author of D3)

viewof bill_length_min = Inputs.range(
  [32, 50], 
  {value: 35, step: 1, label: "Bill length (min):"}
)
viewof islands = Inputs.checkbox(
  ["Torgersen", "Biscoe", "Dream"], 
  { value: ["Torgersen", "Biscoe"], 
    label: "Islands:"
  }
)

• Plot
• Data

Plot.rectY(filtered, 
  Plot.binX(
    {y: "count"}, 
    {x: "body_mass", fill: "species", thresholds: 20}
  ))
  .plot({
    facet: {
      data: filtered,
      x: "sex",
      y: "species",
      marginRight: 80
    },
    marks: [
      Plot.frame(),
    ]
  }
)

Inputs.table(filtered)

data = FileAttachment("palmer-penguins.csv").csv({ typed: true })

filtered = data.filter(function(penguin) {
  return bill_length_min < penguin.bill_length &&
         islands.includes(penguin.island);
})

Interactivity, on the fly Observable “widgets”

Quarto including Observable means you can create new “widgets” or allow the user to modify portions of the doc on the fly.

viewof temp = Inputs.range([0, 100], {step: 1, value: 34, label: htl.html`Temp ℃`})

Converting temperature from ℃ to ℉

Celsius = ℃ and Fahrenheit = ℉.

```{ojs}
viewof temp = Inputs.range([0, 100], {step: 1, value: 34, label: htl.html`Temp ℃`})
```

Converting temperature from &#x2103; to &#x2109; <br>  
Celsius = ${d3.format(".0f")(temp)}&#x2103; and Fahrenheit = ${d3.format(".1f")(temp * 9/5 + 32)}&#x2109;.

ShinyLive + Quarto = WASM

#| standalone: true
#| viewerHeight: 420
#| echo: false
#| eval: true

from shiny import App, render, ui
import numpy as np
import matplotlib.pyplot as plt

app_ui = ui.page_fluid(
    ui.layout_sidebar(
        ui.panel_sidebar(
            ui.input_slider("period", "Period", 0.5, 4, 1, step=0.5),
            ui.input_slider("amplitude", "Amplitude", 0, 2, 1, step=0.25),
            ui.input_slider("shift", "Phase shift", 0, 2, 0, step=0.1),
        ),
        ui.panel_main(
            ui.output_plot("plot"),
        ),
    ),
)


def server(input, output, session):
    @output
    @render.plot(alt="Sine wave")
    def plot():
        t = np.arange(0.0, 4.0, 0.01)
        s = input.amplitude() * np.sin(
            2 * np.pi / input.period() * (t - input.shift() / 2)
        )
        fig, ax = plt.subplots()
        ax.set_ylim([-2, 2])
        ax.plot(t, s)
        ax.grid()


app = App(app_ui, server)

ShinyLive + Quarto = WASM

No server needed, all in browser!

```{shinylive-python}
#| standalone: true
#| viewerHeight: 420

from shiny import App, render, ui
import numpy as np
import matplotlib.pyplot as plt

app_ui = ui.page_fluid(
    ui.layout_sidebar(
        ui.panel_sidebar(
            ui.input_slider("period", "Period", 0.5, 4, 1, step=0.5),
            ui.input_slider("amplitude", "Amplitude", 0, 2, 1, step=0.25),
            ui.input_slider("shift", "Phase shift", 0, 2, 0, step=0.1),
        ),
        ui.panel_main(
            ui.output_plot("plot"),
        ),
    ),
)


def server(input, output, session):
    @output
    @render.plot(alt="Sine wave")
    def plot():
        t = np.arange(0.0, 4.0, 0.01)
        s = input.amplitude() * np.sin(
            2 * np.pi / input.period() * (t - input.shift() / 2)
        )
        fig, ax = plt.subplots()
        ax.set_ylim([-2, 2])
        ax.plot(t, s)
        ax.grid()


app = App(app_ui, server)

```

Parameters - one source, many outputs

• Showing results for a specific geographic location.
  
• Running a report that covers a specific time period.
  
• Running a single analysis multiple times for different assumptions.

Jupyter Engine

```{python}
#| tags: [parameters]

alpha = 0.1
ratio = 0.1
```

The parameters are available in the top level environment:

```{python}
str(alpha)
```

knitr Engine

---
title: "My Document"
params:
  alpha: 0.1
  ratio: 0.1
---

The parameters are available in the params list:

```{r}
str(params$alpha)
```

Rendering with parameters

To render using different parameters you can pass them on the command line using the -P flag:

quarto render notebook.ipynb -P alpha:0.2 -P ratio:0.3

Alternatively, if you have many parameters you can create a YAML file that defines the parameter values you want to render with, then call quarto render from the command line with the --execute-params flag:

quarto render notebook.ipynb --execute-params params.yml

Quarto Publish

quarto publish --help

  Usage:   quarto publish [provider] [path]
  Version 1.4.553                          
                                           
  Description:
    Publish a document or project. Available providers include:
                                                               
     - Quarto Pub (quarto-pub)                                 
     - GitHub Pages (gh-pages)                                 
     - Posit Connect (connect)                               
     - Netlify (netlify)                                       

Built-in vs custom

One goal of Quarto is to provide a markdown-centric format-agnostic syntax as shown in previous slides.

• Quarto bundles Bootstrap CSS and themes, and respects SASS variables for robust styling of HTML content (HTML documents, websites, books, slides, etc).
• Quarto includes LaTeX templates for specific journals as well as good defaults for PDF outputs in general.
• Quarto respects docx and pptx templates, again allowing for robust styling.

• You shouldn’t HAVE to escape out to writing raw LaTeX, HTML, Jinja templates, etc
• In vast majority of situations, can rely purely on Markdown syntax

• BUT you can always include raw content such as LaTeX, CSS, HTML, JavaScript to further customize and optimize for a specific format.

Extending Quarto with extensions

Shortcodes

• Replace inline “short codes” with output.

{{< fa thumbs-up >}} 

Filters

• Affect rendering of specific items

Formats

• Add entirely custom new formats

---
title: "Cool Company 2022 Presentation"
format: coolco-revealjs
---

Quarto, crafted with love and care

Development of Quarto is sponsored by Posit, PBC (formerly known as RStudio, PBC). The same core team works on both Quarto and R Markdown:

• Carlos Scheidegger (@cscheid)

• Charles Teague (@dragonstyle)

• Christophe Dervieux (@cderv)

• J.J. Allaire (@jjallaire)

Here is the full contributors list. Quarto is open source and we welcome contributions in our github repository as well! https://github.com/quarto-dev/quarto-cli.

Quarto

• Batteries included, shared syntax across output types and languages
• Single source publishing across document types, with raw customization allowed
• Choose your own editor for plain text .qmd or Jupyter notebooks
• Quarto projects + freeze for managing stored computation

Follow @quarto_pub #QuartoPub on Twitter/Fosstodon to stay up to date!

Web resources

• Quarto.org for documentation or the Quarto FAQ
• Get Started with Quarto + Python

Quarto resources

• Quarto for the Curious - 20 min video
• ‘Welcome to Quarto’ workshop - 120 min video

General Quarto

• Hello Quarto - RStudio::Conf Keynote, 60 min
• Literate Programming With Jupyter Notebooks and Quarto - 20 min

Why the name “Quarto”?¹

1. Why Quarto? We wanted to use a name that had meaning in the history of publishing and landed on Quarto, which is the format of a book or pamphlet produced from full sheets printed with eight pages of text, four to a side, then folded twice to produce four leaves. The earliest known European printed book is a Quarto, the Sibyllenbuch, believed to have been printed by Johannes Gutenberg in 1452–53.