Formal Interactive Notebook Testing

This post is for notebook authors who need to ship software, and anyone who wants to start learning more about testing in Python. We'll focus on Python's standard library unit testing tools interactively from the notebook. There are many flavors of testing, but our focus remains on unit testing to provide quantitative metrics about the fitness of program. Readers will leave understanding how to prototype and simulate writing unit tests in interactive notebooks.

A common motivation for using notebooks is to test an idea. Without formal conventions, notebooks can result in scatter-shot code that informally verifies an idea. In this post, we discuss how to mature informal notebooks into formal unit test conventions. With practice, an effective use of the notebook is to compose both code and formal tests that can be moved into your project's module and testing suite; you do have tests don't you?

Why Are Formal Tests Valuable

Tests are investments, and testing over time measures the return on investment. Testing promotes:

  • longevity of ideas

  • protection from upstream changes

  • value to you and consumers of your software

  • health metrics when used in continuous deployment

Learn more about the motivation for testing The Hitchhiker's Guide to Python - Testing your code

Testing is Standard

Most programming languages come with unit testing abilities that allow authors to make formal assertions about the expectations of their code. In Python, doctest and unittest are built-in libraries that enable testing; meanwhile, pytest is the choice of popular projects in the broader Python community. You will not need extra dependencies besides a notebook interface and Python to apply the ideas from this post.

We will not discuss testing notebooks in pytest in this document, but if you want to read ahead you can look at nbval, importnb, or testbook for different flavors of notebook testing.

Python Doctest

Documentation-driven testing was