Tensorflow

Tensorboard

Tensorboard is a de facto debugging tool standard. It’s not immediately intuitive; I recommend reading Li Yin’s explanation.

Minimally,

tensorboard --logdir=path/to/log-directory

or, more usually,

tensorboard --logdir=name1:/path/to/logs/1,name2:/path/to/logs/2 --host=localhost

or, lazily,

tensorboard --logdir=$(ls -dm *.logs |tr -d ' \n\r') --host=localhost

In fact, that sometimes works not so well for me. Tensorboard reeeeally wants you to explicitly specify your folder names.

#!/bin/env python3
from pathlib import Path
from subprocess import run
import sys

p = Path('./')

logdirstring = '--logdir=' + ','.join([
    str(d)[:-5] + ":" + str(d)
    for d in p.glob('*.logs')
])

proc = run(
    [
        'tensorboard',
        logdirstring,
        '--host=localhost'
    ]
)

• Projector visualises embeddings:

  TensorBoard has a built-in visualizer, called the Embedding Projector, for interactive visualization and analysis of high-dimensional data like embeddings. It is meant to be useful for developers and researchers alike. It reads from the checkpoint files where you save your tensorflow variables. Although it’s most useful for embeddings, it will load any 2D tensor, potentially including your training weights.

Official documentation

The Tensorflow RNN documentation, as bad as it is, is not even easy to find, being scattered across several non-obvious locations without consistent crosslinks.

• Overview docs
• Other docs of confusing relation to the prior docs
• tutorial docs.

To make it actually make sense without unwarranted time wasting and guessing, you will then need to read other stuff:

Community guides

• seq2seq models with GRUs : Fun with Recurrent Neural Nets.
• Variable sequence length HOWTO.
• Where do the RNN weights come from? Magic.
• Denny Britz’s blog posts * RNNs in Tensorflow, a practical guide and undocumented features. * He also gives a good explanation of vanishing gradients.
• Danijar Hafner * Introduction to Recurrent Networks in TensorFlow * Variable sequence lengths HOWTO
• Philippe Remy, Stateful LSTM in Keras
• Ben Bolte, Deep Language Modeling for Question Answering using Keras

Useful libraries

recurrentshop makes it easier to manage recurrent topologies using keras.

Gradient tricks

Hessian matrix. There is a discussion on Jacobians in TF, including some fancy examples by jjough:

here’s mine — works for high-dimensional Jacobians (numerator and denominator have >1 dimension), undefined batch sizes, and tensors that are not statically known.

Remember to use an interactive session, otherwise tf.get_default_session() will not be able to find the session.

def tf_jacobian(tensor2, tensor1, feed_dict, sess = tf.get_default_session()):
    """
    Computes the tensor d(tensor2)/d(tensor1) recursively.
    :param tensor2: numerator of Jacobian
    :param tensor1: denominator of Jacobian
    :param feed_dict: input data (need this if tensors are not statically known)
    :return: a tensor of dimension (dim_tensor2 x dim_tensor1)
    """
    # can't do tensor.get_shape() because it doesn't work for undefined batch size
    shape = list(sess.run(tf.shape(tensor2), feed_dict))
    if shape:
        # split tensor2 along first dimension and recur
        # int trick from https://github.com/tensorflow/tensorflow/issues/7754
        tensor2_split = tf.split(axis = 0, num_or_size_splits = int(shape[0]), value = tensor2)
        grad_split = [tf_jacobian(tf.squeeze(M, squeeze_dims = 0), tensor1, feed_dict) for M in tensor2_split]
        return tf.stack(grad_split)
    else:
        # calculate gradient of scalar
        grad = tf.gradients(tensor2, tensor1)
        if grad[0] != None:
            return tf.squeeze(grad, squeeze_dims = [0])
        else:
            # replace any undefined gradients with zeros
            return tf.zeros_like(tensor1)

And here’s one for batched tensors:

def batch_tf_jacobian(tensor2, tensor1, feed_dict, sess = tf.get_default_session()):
    """
    Computes the matrix d(tensor2)/d(tensor1) recursively.
    Tensorflow doesn't really have its own Jacobian operator
    (tf.gradients sums over all dims of tensor2).

    :param tensor2: numerator of Jacobian, first dimension is batch
    :param tensor1: denominator of Jacobian, first dimension is batch
    :param feed_dict: input data (need this if tensors are not statically known)
    :return: batch Jacobian tensor
    """
    shape2 = list(sess.run(tf.shape(tensor2), feed_dict))
    shape1 = list(sess.run(tf.shape(tensor1), feed_dict))

    jacobian = tf_jacobian(tensor2, tensor1, feed_dict)
    batch_size = shape2[0]

    batch_jacobian = [tf.slice(jacobian, [i] + [0]*(len(shape2)-1) + [i] + [0]*(len(shape1)-1),  [1] + [-1]*(len(shape2)-1) + [1] + [-1]*(len(shape1)-1)) for i in range(batch_size)]
    batch_jacobian = [tf.squeeze(tensor, squeeze_dims = (0, len(shape2))) for tensor in batch_jacobian]
    batch_jacobian = tf.stack(batch_jacobian)
    return batch_jacobian

Optimisation tricks

Using traditional/experimental optimisers rather than SGD-type ones.