A Swiss army knife of coding tools. Good matrix library, general scientific tools, statistics tools, web server, art tools, but, most usefully, interoperation with everything else - It wraps C, C++, Fortran, includes HTTP clients, parsers, API libraries, and all the other fruits of a thriving community. Fast enough, easy to debug, garbage-collected. If some bit is too slow, you compile it, otherwise, you relax. An excellent choice if you’d rather get stuff done than write code.
Of course, it could be better. clojure is more elegant, scala is easier to parallelise, julia prioritises scientific higher…
But in terms of using a damn-well-supported language that goes on your computer right now, and requires you to reinvent few wheels, and which is transferrable across number crunching, web development, UIs, text processing, graphics and sundry other domains, and does not require heavy licensing costs, this one is a good default choice.
- Asynchrony in python
- Watching files for changes
- ipython, the interactive python upgrade
- Which Foreign Function Interface am I supposed to be using now?
- Which command line parser is the good one?
- Miscellaneous stuff I always need to look up
- Packaging and environments
- Python 2 vs 3
- Misc recommendations
Does this inotify solution work for non-linux? Because OSX uses FSEvents and windows uses I-don’t-even-know.
The python-specific part of jupyter, which can also run without jupyter. Long story.
The main thing I forget here is
Let’s say there is a line in your code that fails:
In vanilla python if you want to debug the last exception (the post-mortem debugger) you do:
import pdb; pdb.pm()
and if you want to drop into a debugger from some bit of code, you write:
import pdb; pdb.set_trace()
and if you want to use a fancier debugger (ipdb is recommended):
import ipdb; ipdb.set_trace()
import ipdb; ipdb.pm()
This doesn’t work in jupyter, which has some other fancy interaction loop going on.
Here’s one manual way to drop into the debugger from code, noticed by Christoph Martin
from IPython.core.debugger import Tracer; Tracer()() 1/0
%debug [--breakpoint filename:line_number_for_breakpoint]
Without the argument it activates post-mortem mode. Seriously though, who thinks in line-numbers? Tracer realistically wastes less time.
Update: as of ipython 5.1, this is the new recommended way:
from IPython.core.debugger import set_trace; set_trace()
And if you want to drop automatically into the post mortem debugger for every error:
%pdb on 1/0
Gaël recommended some extra debuggers:
- aiomonitor is repl-injection for async python
- pudb, a curses-style debugger, is very popular.
- The trepan family of debuggers, trepan3k (python 3), trepan (python 2), ipython-trepan (theoretically ipython but looks unmaintained). Docs live here/.
- My brother Andy likes the intellij IDE’s built-in python debugger.
- There are many other debuggers.
But maybe it’s not crashing but taking too long? You want a profiler.
import cProfile as profile profile.runctx('print(predded.shape)', globals(), locals())
- h(elp) [command]
- Print your location in current stack
- d(own) [count]/up [count]
- Move the current frame count (default one) levels down/ in the stack trace (to a newer frame).
- b(reak) [([filename:]lineno | function) [, condition]]
- The one that is tedious to do manually. Without argument, list all breaks and their metadata.
- tbreak [([filename:]lineno | function) [, condition]]
- Temporary breakpoint, which is removed automatically when it is first hit.
- cl(ear) [filename:lineno | bpnumber [bpnumber ...]]
- Clear specific or all breakpoints
- disable [bpnumber [bpnumber ...]]/enable [bpnumber [bpnumber ...]]
- disable is the same as clear, but you can re-enable
- ignore bpnumber [count]
- ignore a breakpoint a specified number of times
- condition bpnumber [condition]
- Set a new condition for the breakpoint
- commands [bpnumber]
- Specify a list of commands for breakpoint number bpnumber. The commands themselves appear on the following lines. Type end to terminate the command list.
- Execute the next line, even if that is inside an invoked function.
- Execute the next line in this function.
- unt(il) [lineno]
- continue to line lineno, or the next line with a highetr number than the current one
- Continue execution until the current function returns.
- Continue execution, only stop when a breakpoint is encountered.
- j(ump) lineno
- Set the next line that will be executed. Only available in the bottom-most frame. It is not possible to jump into weird places like the middle of a for loop.
- l(ist) [first[, last]]
- List source code for the current file.
- ll | longlist
- List all source code for the current function or frame.
- Print the argument list of the current function.
- p expression
- Evaluate the expression in the current context and print its value.
- pp expression
- Like the p command, except the value of the expression is pretty-printed using the pprint module.
- whatis expression
- Print the type of the expression.
- source expression
- Try to get source code for the given object and display it.
- display [expression]/undisplay [expression]
- Display the value of the expression if it changed, each time execution stops in the current frame.
- Start an interactive interpreter (using the code module) whose global namespace contains all the (global and local) names found in the current scope.
- alias [name [command]]/unalias name
Create an alias called name that executes command.
As an example, here are two useful aliases from the manual, for the .pdbrc file:
# Print instance variables (usage ``pi classInst``) alias pi for k in %1.__dict__.keys(): print("%1.",k,"=",%1.__dict__[k]) # Print instance variables in self alias ps pi self
- ! statement
- Execute the (one-line) statement in the context of the current stack frame, even if it mirrors the name of a debugger command
- Pack up and go home
Check out the Rich display protocol which allows you to render objects as arbitrary graphics.
How to use this? The display api docs explain that you should basically implement methods such as _repr_svg_.
I made a thing called latex_fragment which leverages this to display arbitrary latex inline.
def _figure_data(self, format): fig, ax = plt.subplots() ax.plot(self.data, 'o') ax.set_title(self._repr_latex_()) data = print_figure(fig, format) # We MUST close the figure, otherwise IPython's display machinery # will pick it up and send it as output, resulting in a double display plt.close(fig) return data # Here we define the special repr methods that provide the IPython display protocol # Note that for the two figures, we cache the figure data once computed. def _repr_png_(self): if self._png_data is None: self._png_data = self._figure_data('png') return self._png_data
For a non-graphical non-fancy terminal, you probably simply want nice formatting of dictionaries:
from pprint import pprint, pformat pprint(obj) # display it print(pformat(obj)) # get a nicey formatted representation
Sometimes it’s just complicated to work out how to load some complicated dependency tree of stuff. Use deepreload. You can even hack traditional reload to be deep.
import builtins from IPython.lib import deepreload builtins.reload = deepreload.reload
There is also an autoreload extension which in principle reloads everything that has changed.
%load_ext autoreload %autoreload 2
If you load them both at the same time, stuff gets weird. Don’t do that.
Profile functions using cProfile.
Now visualise them using… uh…
Want to call a a function in C+, C++, FORTRAN etc from python? Possibly to go faster?
And of course, if you have your compiler lying about, Python was made to talk to other languages and has (has always had) a normal C API.
If you want something closer to python for you development process, Cython allows some python compilation and easy calling of foreign functions. SWIG wraps function interfaces between various languages, but looks like a PITA; (See a comparison on stackoverflow).
There is also Boost.python if you want to talk to C++.
argparse is built-in to python stdlib and is completely fine.
argh was/is a popular extension to argparse
Argh is fully compatible with argparse. You can mix Argh-agnostic and Argh-aware code. Just keep in mind that the dispatcher does some extra work that a custom dispatcher may not do.
Click is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary. It’s the “Command Line Interface Creation Kit”. It’s highly configurable but comes with sensible defaults out of the box.[…]
- arbitrary nesting of commands
- automatic help page generation
- supports lazy loading of subcommands at runtime
Not so hard, but confusing and chaotic due to many long-running disputes only lately resolving.
- Least nerdview guide ever: Vicki Boykis, Alice in Python projectland.
- Simplest readable guide is python-packaging
- Sharing Your Labor of Love: PyPI Quick and Dirty, includes some good tips such as using twine to make it automaticker.
- Open-sourcing a python project the right way.
- Official docs are no longer awful but are slightly stale, and especially perfunctory for compilation
- Kenneth Reitz shows rather than tells with a heavily documented example
- Zed Shaw’s signature aggressively cynical and reasonably practical explanation of project structure, with bonus explication of how you should expect much time wasting yak shaving like this if you want to do software.
The distribution you use if you want to teach a path in numerical python without dicking around with a 5 hour install process.
The upshot is if you want to install something with tricky dependencies like ViTables, you do this:
conda install pytables=3.2 conda install pyqt=4
source (conda info —root)/etc/fish/conf.d/conda.fish
venv is now a builtin virtual python environemtn system in python 3. It doesn’t support python 2 but fixes various problems, e.g. it supports framework python on OSX which is very important for GUIs, and is covered by the python docs in the python virtual environment introduction.
# Create venv python3 -m venv ~/.virtualenvs/learning_gamelan_keras_2 # Use venv from fish source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate.fish # Use venv from bash source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate
Too many bike sheds.
Jacon Kaplan-Moss likes pytest and he’s good let’s copy him. FWIW I’m no fan of nose; my experience of it was that I spent a lot of time debugging weird failures getting lost in its attempts to automagically help me. This might be because I didn’t deeply understand what i was doing, but the other frameworks didn’t require me to understand that deeply the complexities of their attempts to simplify my life.
Hypothesis is a library which does randomised constraint-based testing:
It works by generating random data matching your specification and checking that your guarantee still holds in that case. If it finds an example where it doesn’t, it takes that example and cuts it down to size, simplifying it until it finds a much smaller example that still causes the problem.
- Sebastian Raschka, The key differences between Python 2.7.x and Python 3.x with examples
- Writing python 2 and 3 compatible code
TODO: six versus future.
Python 3.6 includes type hinting, and projects such as mypy support static analysis using type hints. There are not yet many tutorials on the details of this, but for once tutsplus has one of the better ones.
Short version: you go from this:
def fib(n): a, b = 0, 1 while a < n: yield a a, b = b, a+b
def fib(n: int) -> Iterator[int]: a, b = 0, 1 while a < n: yield a a, b = b, a+b
which looks like a great idea from where I’m sitting.
However, if you are going to this trouble, why not just use julia, which will take your type hints and compile them into fast code?