The Living Thing / Notebooks :

Python

A programming language whose remarkable and rare feature is working like you imagine, if not how it should

A Swiss army knife of coding tools. Good matrix library, general scientific tools, statistics tools, web server, art tools, but, most usefully, interoperation with everything else - It wraps C, C++, Fortran, includes HTTP clients, parsers, API libraries, and all the other fruits of a thriving community. Fast enough, easy to debug, garbage-collected. If some bit is too slow, you compile it, otherwise, you relax. A good default if you’d rather get stuff done than write code.

I typically do my stats and graphs in R, my user interface in javascript, my linear algebra library is fortran, and I use julia for my other scientific calculations, but python is the thread that stitches this Frankensteinisch monster together.

Of course, it could be better. clojure is more elegant, scala is easier to parallelise, julia prioritises science more highly….

But in terms of using a damn-well-supported language that goes on your computer right now, and requires you to reinvent few wheels, and which is transferrable across number crunching, web development, UIs, text processing, graphics and sundry other domains, and does not require heavy licensing costs, this one is a good starting point.

What a pitch! Now, let’s look a bit closer and see all the horrid things that are wrong with it.

Debugging, profiling and testing

See Python debugging, profiling and testing.

Pro tip: dotenv

dotenv allows eaasy configuration thourgh OS environemtne variables or text files in the parent directory. You should probably use this. (Careful, there are lots of packages with similar names.)

python pip install python-dotenv

String formatting

String formatting things that are unnecessarily hard to discern from the manual

What a nightmare is that manual for the string formatting. While all the information you need is in there, it is arranged in perverse inversion of some mixture of the frequency and the priority with which you use it. See Marcus Kazmierczak’s cookbook instead.

Highlights:

## float precision
>>> print("{:.2f}".format(3.1415926))
3.14
## left padding
>>> print("{:0>2d}".format(5))
05
## power tip which the manual does not make clear:
## variable formatting
>>> pi = 3.1415926
>>> precision = 4
>>> print( "{:.{}f}".format( pi, precision ) )
3.1415

Rendering HTML output

You have a quick sand dirty but of HTML you need to output? You aren’t writing some dmaned webapp with a nested hierarchy of tempalte rendering and CSS integration into some design framework? you just want to pump out some markup?

I recommend yattag which is fast, simple, good and has a 1-page manual. It works real good.

Rendering markdown as HTML

… code:: python

from markdown import markdown html_string = markdown(“”“# Title Paragraph. emphasis. “”“)

Which Foreign Function Interface am I supposed to be using now?

Want to call a a function in C+, C++, FORTRAN etc from python?

If you are just talking to C, ctypes is a python library to translate python objects to c with minimal fuss, and no compiler requirement. See the ctypes tutorial.

And of course, if you have your compiler lying about, Python was made to talk to other languages and has a normal C API.

If you want something closer to python for your development process, Cython allows python compilation using a special syntax, and easy calling of foreign functions in one easy package. SWIG wraps function interfaces between various languages, but looks like a PITA; (See a comparison on stackoverflow).

There is also Boost.python if you want to talk to C++. Boost comes with lots of other fancy bits, like numerical libraries.

There are many other options, but in practice I’ve never needed to go further than cython, so I can’t even talk about all the options listed here knowledgeably.

Packaging and environments

Not so hard, but confusing and chaotic due to many long-running disputes only lately resolving.

General

Anaconda

The distribution you use if you want to teach a course in numerical python without dicking around with a 5 hour install process.

Has a slightly different packaging workflow. See, e.g. Tim Hoppper’s workflow which explains this environment.yml malarkey, or the creators’ rationale.

The upshot is if you want to install something with tricky dependencies like ViTables, you do this:

conda install pytables=3.2
conda install pyqt=4

Aside: Do you use fish shell? You need to do some extra setup. Specifically, add the line

source (conda info --root)/etc/fish/conf.d/conda.fish

into ~/.config/fish/config.fish.

NB Conda will fill up your hard disk if not regularly disciplined. via conda clean to stop your disk filling up with obsolete versions of obscure dependencies for that package you tried out that one time.

conda clean -pt

You might also want to not have the gigantic MKL library installed, of which I am in any case not a fan. You can disable it per request:

conda create -n pynomkl python nomkl

Note that the packagers claim this is a 100MB library. Poppycock. The package alone is 800MB, and it’s even bigger installed. That’s not counting the fact that conda will keep many many versions around. MKL alone was using about 10GB total no my machine when I last checked, which is two orders of magnitude off what the unwarey might assume reading that manual page.

Reducing the harm of this kind of nonsense is one reason you should only ever install the minimalist miniconda as your base anaconda distribution, otherwise you might be tempted to try to use the base environemnt to do something, which will lead to even more bloat.

Python environment/version management

venv is now a builtin virtual python environemnt system in python 3. It doesn’t support python 2 but fixes various problems, e.g. it supports framework python on OSX which is very important for GUIs, and is covered by the python docs in the python virtual environment introduction.

# Create venv
python3 -m venv ~/.virtualenvs/learning_gamelan_keras_2
# Use venv from fish
source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate.fish
# Use venv from bash
source ~/.virtualenvs/learning_gamelan_keras_2/bin/activate

Python environment management management

One suggestion I’ve has is to use pyenv. which eases and automates switching between all the other weird python environments created by virtualenv, python.org python, os python, anaconda python etc.

BUT WHO MANAGES THE VIRTUALENV MANAGER MANAGER?

Asynchrony in python

See asynchronous python

Watching files for changes

Does this inotify solution work for non-linux? Because OSX uses FSEvents and windows uses I-don’t-even-know.

watchdog asserts that it is cross-platform. (source)

Python 2 vs 3

TODO: six versus future.

TLDR: I am no employee of giant enterprise-type business with a gigantic legacy code base, and so I don’t use python 2. My code is not python 2 compatible. Python 3 is more productive, and no-one is paying me to be less productive right now. Python 2 code is usually easy to port to python 3. It is possible to write code which is compatible with python 2 and 3, but then I would miss out on lots of the work that has gone into making python 3 easier and better, and waste time porting elegant easy python 3 things to hard boring python 2 things.

Typing

Python 3.6 includes type hinting, and projects such as mypy support static analysis using type hints. There are not yet many tutorials on the details of this. Here’s one.

Short version: you go from this:

def fib(n):
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a+b

to this:

def fib(n: int) -> Iterator[int]:
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a+b

However, if you are going to this trouble, why not just use julia, which takes type-hinting a lot futher and will actualy use it to JIT-compile optimized code?

Which command line parser is the good one?

Another bike-shedding danger zone is command-line parsing, leading to the need to spend too much time parsing command line parsers rahter than parsing command lines.

Argh is fully compatible with argparse. You can mix Argh-agnostic and Argh-aware code. Just keep in mind that the dispatcher does some extra work that a custom dispatcher may not do.

Miscellaneous stuff I always need to look up

Misc recommendations