I have written a few simple keras layers. This post will summarise about how to write your own layers. It’s for beginners because I only know simple and easy ones 😉
1. Keras layer
https://keras.io/layers/about-keras-layers/ introduces some common methods. For beginners I don’t think it’s necessary to know these.
2. Keras Lambda layer
Lambda layer is an easy way to customise a layer to do simple arithmetics. As written in the page,
…an arbitrary Theano / TensorFlow expression…
we can use the operations supported by Keras backend such as
dot, transpose, max, pow, sign, etc as well as those are not specified in the backend documents but actually supported by Theano and TensorFlow – e.g.,
**, /, //, % for Theano.
2.1 Lambda layer and output_shape
You might need to specify the output shape of your Lambda layer, especially your Keras is on Theano. Otherwise it just seems to infer it with
2.1.1 With function
You can create a function that returns the output shape, probably after taking input_shape as an input. Here, the function returns the shape of the WHOLE BATCH.
2.1.2 With tuple
If you pass tuple, it should be the shape of ONE DATA SAMPLE.
3. A Keras model as a layer
On high-level, you can combine some layers to design your own layer. For example, I made a Melspectrogram layer as below. (Complete codes are on keras_STFT_layer repo.) In this way, I could re-use
Convolution2D layer in the way I want.
Downside would be some overhead due to many layers.
4. Customising Layer
When Lego-ing known layers doesn’t get you what you want, write your own!
4.1 Read the document
https://keras.io/layers/writing-your-own-keras-layers/ Read this! Whether you fully understand it or not. I didn’t fully understand but later I got it thanks to @fchollet’s help.
4.2 Four methods
initiate the layer. Assign attributes to
self so that you can use them later.
build(self, input_shape) :
- initiate the tensor variables (e.g.
bias, or whatever) using Keras backend functions (e.g.,
self.W = K.variable(an_init_numpy_array)).
self.trainable_weightswith a list of variables. e.g.,
Remember : trainable weights should be tensor variables so that machine can auto-differenciate them for you.
Remember (2): Check out the dtype of every variable! If you initiate a tensor variable with
float64 a numpy array, the variable might be also
float64, which will get you an error. Usually it wouldn’t because by default
K.variable() casts the value into
float32. But, check check check! check it by simply printing
call(self, x, mask=None) :
This is where you implement the forward-pass operation. You may want to dot product with one of the trainable weights and input (
K.dot(x, self.W)), wanna expand the dimensionality of a tensor variable (
K.expand_dims(var1, dim=2)), or whatever.
Again, dtype! For example, I had to use this line,
np.sqrt(2. * np.pi).astype('float32'), to make the constant to be
As the name says.
4.3.1 Example 1 : Cropping2D Layer
It crops 2D input. Simple!
4.3.2 Example 2. ParametricMel
Remember: you need to make the operation of layer differentiable w.r.t the input and trainable weights you set. Look up keras backend use them.
tensor.shape.eval() returns an integer tuple. You would need to print them a lot 😉