Python debugging, profiling and testing

3.1 Built-in debugger

Let’s say there is a line in my code that fails:

1/0

In vanilla python if I want to debug the last exception (the post-mortem debugger) I do:

import pdb; pdb.pm()

and if I want to drop into a debugger from some bit of code, I write:

import pdb; pdb.set_trace()

or in python 3.7+:

breakpoint()

This is a pretty good solution that works well and is available AFAICT everywhere. The main problem is that they constantly change the recommended way of invoking the debugger. Get ready for a LONG LIST OF ALTERNATIVES.

If I want a debugger with rich autocomplete, there is a nice one in ipython. Here’s a manual way to drop into the ipython debugger from code, according to Christoph Martin and David Hamann:

from IPython.core.debugger import Tracer; Tracer()()  # < 5.1
from IPython.core.debugger import set_trace; set_trace()  # >= v5.1

However, that’s not how we are supposed to do it in polite society. Persons of quality are rumoured to invoke their debuggers via so-called magics, e.g. the %debug magic to set a breakpoint at a certain line number:

%debug [--breakpoint filename:line_number_for_breakpoint]

Pish posh, who thinks in line-numbers? set_trace wastes less time for humans per default.

An actual use I would make of %debug is to drop into post-mortem debugging; Without the argument, %debug activates post-mortem mode. And if I want to drop automatically into the post mortem debugger for every error:

%pdb on

Props to Josh Devlin for explaining this and some other handy tips, and also Gaël Varoquaux.

If that seems abstruse or verbose, ipdb exposes the enhanced debugger from ipython simply and explicitly:

pip install ipdb

import ipdb; ipdb.set_trace()

or:

import ipdb; ipdb.pm()

ipdb doesn’t work in jupyter, whose interaction loop is incompatible. %debug does, but it’s fairly horrible, because juptyer frontends are a mess and various things break; e.g. if I try to execute non-debugger code while in the debugger the entire notebook sometimes freezes unrecoverably; this is very easy to do because the debug console is small and easy to miss when trying to click on it. Any time I find myself needing to debug debugging in jupyter I am briefly filled with despair, then I remember that there is no overwhelming moral imperative for me to use jupyter for anything and I can switch to ipython or vs code.

4.1 VS Code debugger

See VS Code python debugger.

4.2 pudb

Pudb seems to be very close to the native debugger but with console enhancements

• Syntax-highlighted source, the stack, breakpoints and variables are all visible at once and continuously updated. This helps you be more aware of what’s going on in your program. Variable displays can be expanded, collapsed and have various customization options.
• Simple, keyboard-based navigation using single keystrokes makes debugging quick and easy. PuDB understands cursor-keys and Vi shortcuts for navigation. Other keys are inspired by the corresponding pdb commands.
• Drop to a Python shell in the current environment by pressing “!”. Or open a command prompt alongside the source-code via “Ctrl-X”.
• Ability to control the debugger from a separate terminal.

• pudb 2021.1 documentation
• Debug Python With Pudb

4.3 PyCharm

My brother Andy likes the PyCharm/IntelliJ IDE’s built-in python debugger. I have not used it.

4.4 Viztracer

VizTracer

… is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code to help you intuitively understand your code and figure out the time consuming part of your code.

VizTracer can display every function executed and the corresponding entry/exit time from the beginning of the program to the end, which is helpful for programmers to catch sporatic (sic) performance issues.

Sure, sounds fine.

4.5 pysnooper

PySnooper claims:

instead of carefully crafting the right print lines, you just add one decorator line to the function you’re interested in. You’ll get a play-by-play log of your function, including which lines ran and when, and exactly when local variables were changed.

I always think I’d like to use this, but in practice I don’t.

4.6 Pyrasite

pyrasite injects code into running python processes, which enables more exotic debuggery, and realtime object mutation and stuff and of course, memory and performance profiling.

4.7 Yet more

Gaël recommended some extra debuggers:

• aiomonitor is REPL-injection for async python
• pudb, a curses-style debugger, is popular.
• The trepan family of debuggers, trepan3k (python 3), trepan (python 2), ipython-trepan (theoretically ipython but looks unmaintained). Docs live here.

Jeez, OK. But wait there are more.

• There are many other debuggers.
• That’s too many debuggers
• Realistically I won’t use any of them, because the inbuilt one is OK, and already hard enough to keep in my head without putting more points of failure in the mix
• Stop making debuggers

5.1 Memray

Memory specialist. bloomberg/memray: Memray is a memory profiler for Python

Memray is a memory profiler for Python. It can track memory allocations in Python code, in native extension modules, and in the Python interpreter itself. It can generate several different types of reports to help you analyze the captured memory usage data. While commonly used as a CLI tool, it can also be used as a library to perform more fine-grained profiling tasks.

Notable features:

• 🕵️‍♀️ Traces every function call so it can accurately represent the call stack, unlike sampling profilers.
• ℭ Also handles native calls in C/C++ libraries so the entire call stack is present in the results.
• 🏎 Blazing fast! Profiling slows the application only slightly. Tracking native code is somewhat slower, but this can be enabled or disabled on demand.
• 📈 It can generate various reports about the collected memory usage data, like flame graphs.
• 🧵 Works with Python threads.
• 👽🧵 Works with native-threads (e.g. C++ threads in C extensions).

Memray can help with the following problems:

• Analyze allocations in applications to help discover the cause of high memory usage.
• Find memory leaks.
• Find hotspots in code that cause a lot of allocations.

Note that Memray only works on Linux and cannot be installed on other platforms.

5.2 Scalene

See below.

6.1 Built-in profiler

Profile functions using cProfile:

import cProfile as profile
profile.runctx('print(predded.shape)', globals(), locals())

CProfile is not so hip any longer. There are some other ones that are more fashionable.

6.2 Scalene

plasma-umass/scalene

… is a high-performance CPU, GPU and memory profiler for Python that does a number of things that other Python profilers do not and cannot do. It runs orders of magnitude faster than many other profilers while delivering far more detailed information. It is also the first profiler ever to incorporate AI-powered proposed optimizations.

Includes web-gui and VS code integration.

Maybe the freshest thing here? Colleagues of mine love it but I have not used it.

6.3 py-spy

py-spy

[…] lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way. Py-Spy is extremely low overhead: it is written in Rust for speed and doesn’t run in the same process as the profiled Python program, nor does it interrupt the running program in any way. This means Py-Spy is safe to use against production Python code. […]

This project aims to let you profile and debug any running Python program, even if the program is serving production traffic. […]

Py-spy works by directly reading the memory of the python program using the process_vm_readv system call on Linux, the vm_read call on macOS or the ReadProcessMemory call on Windows.

Figuring out the call stack of the Python program is done by looking at the global PyInterpreterState variable to get all the Python threads running in the interpreter, and then iterating over each PyFrameObject in each thread to get the call stack.

Native ipython can run profiler magically:

%%prun -D somefile.prof
files = glob.glob('*.txt')
for file in files:
  with open(file) as f:
    print(hashlib.md5(f.read().encode('utf-8')).hexdigest())

Great worked example — Making Python 100x faster with less than 100 lines of Rust:

Python has a built in Profiler (cProfile), but in this case it’s not really the right tool for the job:

1. It’ll introduce a lot of overhead to all the Python code, and none for native code, so our results might be biased.
2. We won’t be able to see into native frames, meaning we aren’t going to be able to see into our Rust code.

We are going to use py-spy (GitHub).

py-spy is a sampling profiler which can see into native frames.

They also mercifully publish pre-built wheels to pypi, so we can just pip install py-spy and get to work.

6.4 Score-P

HPC-friendly profiling can be provided by scorep a python binding onf the popular multiprocessing score function. Gocht, Schöne, and Frenzel (2021):

In this paper, we present the Python bindings for Score-P, which make it easy for users to trace and profileFootnote 4 their Python applications, including the usage of (multi-threaded) libraries, MPI parallelism and accelerator usage.

• Tool Time: Profiling and Tracing of Python Code with Score-P | Performance Optimisation and Productivity
• VI-HPS :: Projects :: Score-P
• score-p/scorep_binding_python: Allows tracing of python code using Score-P
• Profile and Trace an Application · score-p/scorep_binding_python Wiki

6.5 Austin

I do not know much about this.

• P403n1x87/austin-python: Python wrapper for Austin, the CPython frame stack sampler.
• P403n1x87/austin: Python frame stack sampler for CPython

6.6 Visualising profiles

• flamegraph

• speedscope

• snakeviz is a browser-based system that might be ok for the output of CProfile profiles

• ftrace profiles

  • Chrome’s catapult system can view traces - chrome://tracing/ or brave://tracing/ in the browser
  • They have a new UI called perfetto

• convert the output to cachegrind format for visualisation in the many cachegrind tools.

• py-spy includes built-in flame graphs

• runsnakerun — the original python profiling visualizer, now expired.

SnakeViz includes a handy magic to automatically save stats and launch the profiler. (Gotcha: I have to have the snakeviz CLI already on the path when I launch ipython.)

%load_ext snakeviz
%%snakeviz
files = glob.glob('*.txt')
for file in files:
  with open(file) as f:
    print(hashlib.md5(f.read().encode('utf-8')).hexdigest())

This is incompatible with autoreload and gives weird errors if I run them both in the same session.