Successor to Lua’s torch. Evil twin to googles’s Tensorflow.
They claim to be aiming for fancy features such as reversible learning and other advanced regularisation and what-have-you.
PyTorch has a unique way of building neural networks: using and replaying a tape recorder.
Most frameworks such as TensorFlow, Theano, Caffe and CNTK have a static view of the world. One has to build a neural network, and reuse the same structure again and again. Changing the way the network behaves means that one has to start from scratch. [… Pytorch] allows you to change the way your network behaves arbitrarily with zero lag or overhead.
In fact there is not zero overhead, rather they have shited the overhead baseline down a little. But whatever, it’s really convenient.
There is some bad advice in the manual
nnexports two kinds of interfaces - modules and their functional versions. You can extend it in both ways, but we recommend using modules for all kinds of layers, that hold any parameters or buffers, and recommend using a functional form parameter-less operations like activation functions, pooling, etc.
So, important missing information.
If your desired loss is already just a composition of existing functions, you don’t need to define a
The given options are not a binarism but two things you need to do in concert. A better summary would be:
If you need to have a function which is differentiable in a non-trivial way, implement a Function
If you need to bundle a Function with some state or differentiable parameters, additionally wrap it in a
Some people claim you can also create custom layers using plain python functions. However, these don’t work as layers, in an
nn.Sequentialmodel, so I’m not sure how to take this advice.
It’s just as well it’s easy to roll your own recurrent nets because the default implementations are bad
The default RNN layer is heavily optimised using CUDNN, which is sweet, but you only have a choice of 2 activation functions, and neither of them is “linear”.
Ding Ke made a beautiful and simple RNN implementation.
Utility libraries, derived software
Logging and visualizing training
- Leveraging tensorflow’s handy diagnostic GUI,
tensorboard: tensorboardX or perhaps tensorboard-logger.
Fiddly. The official way is via ONNX.
conda install -c ezyang onnx pydot # or pip install onnx pydot
brew cask install netron # or pip install netron brew install graphviz
Also available, pytorchviz.
pip install git+https://github.com/szagoruyko/pytorchviz
from pytorchviz import make_dot y = model(x) make_dot(y, params = dict(model.named_parameters())
TNT is a reimplementation of some lua library that the current generation of ML users never witnessed. I think it’s semi-official library for pytorch.
TNT (imported as torchnet) is a framework for PyTorch which provides a set of abstractions for PyTorch aiming at encouraging code re-use as well as encouraging modular programming. It provides powerful dataloading, logging, and visualization utilities.[…]
For example, TNT provides simple methods to record model preformance in the torchnet.meter module and to log them to Visdom (or in the future, TensorboardX) with the torchnet.logging
Like other deep learning frameworks, there is some basic NLP support in pytorch; see pytorch.text.
Pump graphs to a visualisation server. No pytorch-specific, but seems well-integrated. visdom
Appears to be a port of the tensorflow keras library to pytorch.
pytorch + Bayes = pyro, an
Pyro launch announcment:
We believe the critical ideas to solve AI will come from a joint effort among a worldwide community of people pursuing diverse approaches. By open sourcing Pyro, we hope to encourage the scientific world to collaborate on making AI tools more flexible, open, and easy-to-use. We expect the current (alpha!) version of Pyro will be of most interest to probabilistic modelers who want to leverage large data sets and deep networks, PyTorch users who want easy-to-use Bayesian computation, and data scientists ready to explore the ragged edge of new technology.
inferno is a grab-bag library for torch.
Current features include:
a basic Trainer class to encapsulate the training boilerplate (iteration/epoch loops, validation and checkpoint creation),
a graph API for building models with complex architectures, powered by networkx.
easy data-parallelism over multiple GPUs,
a submodule for torch.nn.Module-level parameter initialization,
a submodule for data preprocessing / transforms,
[other stuff that is not especially useful to do with a library]
I’m not sold on this one; A whole new library to reduce an already small amount of boilerplate, without adding any new non-trivial capabilities.
Apparently you just use normal python garbage collector analysis.
import torch import gc for obj in gc.get_objects(): try: if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)): print(type(obj), obj.size()) except Exception as e: pass
See also usual python debugging.