Digital scientific workbooks

The exploratory-algorithm-person’s IDE-equivalent. Literate coding-meets-science.

Let’s say I want to demonstrate my algorithm to my thesis advisor while he’s off at conference. I need an easily shareable demonstration. that’s why we have the internet, and scientific computation/documentation system such as Rmarkdown/knitr and Jupyter.

See also text editors, citation management, academic writing workflow, python, cloud computing, open notebook science, scientific computation workflows.

General

Why do this? To belatedly immanentize the prophecy that the scientific paper is dead. That last link is written for outsiders to the field.

If less poetic, Yihui Xie is far more useful: Notebook war summarises some philosophical and practical differences between the literate coding/exploratory notebook hybrid tools in use.

Nextjournal is a collaborative coding machine that claims to make this easy for you and your colleagues to write in a workbook style together.

Jupyter

jupyter is weird enough to have its own notebook. It is slightly python centric but pretty good with multiple languages and AFAICT secretly used inside e.g. RStudio. But it is very odd.

Chris Sewell has produced a script called ipypublish that eases some of the pain points in producing articles starting from jupyter. See the comments for some additional pro-tips for this.

Pweave

Pweave, by Matti Pastell, is python twin to knitr, in the lineage of literate coding tools.

Pweave is a scientific report generator and a literate programming tool for Python. It can capture the results and plots from data analysis and works well with numpy, scipy and matplotlib.

Max Masnick gives a detailed set up example.

Weave.jl

The Julia twin to Weave. There is are native options, weave.jl (See also Literate.jl).

Weave is another literate coding thingy, much like RMarkdown or pweave.

# capturing code output

The code chunk wil be run with default options and the output captured.
<<>>==
using Gadfly
x = linspace(0, 2* pi)
println(x)
plot(x = x, y = sin(x)
@

Or you could use RMarkdown in julia mode. It’s not yet clear to me how graphing works in that case.

knitr/RMarkdown

knitr / RMarkdown are complementary r-based entrants in the scientific workbook field. There are several pieces in this toolchain with a complicated relationship, but the user can ignore most of this complexity. The result is such fanciness as automatically rendering and caching graphs, an interactive notebook UI, nearly first-class support for python and julia plus mediocre support for other languages. Here are some guides:

• interactive reports with R and knitr
• Original author Yihui Xie’s examples gallery has great material for learning-by-cut-and-paste
• His draft knitr manual is free online.

To execute it from the command-line you do

R -e "rmarkdown::render('script.Rmd',output_file='output.html')"

For an intro to the various way to build this into a full reproducible research workflow, see the excellent reproducible analysis workshop.

You can include graphics via markdown native markdown or via R itself. The latter is more powerful if more circuitous.

```{r  out.width = "20%"}
include_graphics(img1_path)
```

Equation references are supported.

See equation \@ref(eq:linear)

\begin{equation}
a + bx = c  (\#eq:linear)
\end{equation}

Some miscellaneous tips:

• How to customise the pandoc output by e.g. adding extensions? There are two ways, Markdown variants and md_extensions option.
• kable (plus possibly kableExtra) is recommended as a multiple-backend table creator.
• pander is recommended for the same purpose; It targets pandoc.
• Might stargazer be useful? It generates marked-up and formatted tables for tabular data, but seems not be generic about output formats.

• tufte is an Edward-Tufte-compliant stylesheet. (cran link)

  tint is an alternate version.

• For online web-friendly (e.g. teaching) purposes, there is literate interactive coding using shiny and latex via shinytex.

MATLAB

• Neil Lawrences’s matlab sweave hacks
• Marking up MATLAB comments for publishing

Editor/IDE support

Miscellaneous preview support scripts are given in the knitr documentation.

RStudio

As noted, RStudio has intimate RMarkdown integration.

Atom

Atom supports a number of literate programming tools via language-weave package - you might also want a full typesetting experience via the latex or atom-latex package, which can be made to support literate coding of plain latex. It uses Hydrogen to provide code preview.

Setting up a latex toolchain in atom-latex is not too bad. E.g. here is one for knitr:

{
    "root": "path/to/my/file.Rnw",
    "toolchain": "Rscript -e \"library(knitr); knit('%DOC.%EXT')\" && latexmk -synctex=1 -interaction=nonstopmode -file-line-error -pdf %DOC",
    "latex_ext": [".Rnw"]
}

It would probably also work for pweave or Weave.jl.