Gating mechanism

In neural networks, the gating mechanism is an architectural motif for controlling the flow of activation and gradient signals. They are most prominently used in recurrent neural networks (RNNs), but have also found applications in other architectures.

RNNs

Gating mechanisms are the centerpiece of long short-term memory (LSTM). They were proposed to mitigate the vanishing gradient problem often encountered by regular RNNs.

An LSTM unit contains three gates:

The equations for LSTM are:

𝐈t=Οƒ(𝐗t𝐖xi+𝐇tβˆ’1𝐖hi+𝐛i)𝐅t=Οƒ(𝐗t𝐖xf+𝐇tβˆ’1𝐖hf+𝐛f)𝐎t=Οƒ(𝐗t𝐖xo+𝐇tβˆ’1𝐖ho+𝐛o)𝐂~t=tanh⁑(𝐗t𝐖xc+𝐇tβˆ’1𝐖hc+𝐛c)𝐂t=𝐅tβŠ™π‚tβˆ’1+𝐈tβŠ™π‚~t𝐇t=𝐎tβŠ™tanh⁑(𝐂t)

Here, βŠ™ represents elementwise multiplication.

The gated recurrent unit (GRU) simplifies the LSTM. Compared to the LSTM, the GRU has just two gates: a reset gate and an update gate. GRU also merges the cell state and hidden state. The reset gate roughly corresponds to the forget gate, and the update gate roughly corresponds to the input gate. The output gate is removed.

There are several variants of GRU. One particular variant has these equations:

𝐑t=Οƒ(𝐗t𝐖xr+𝐇tβˆ’1𝐖hr+𝐛r)𝐙t=Οƒ(𝐗t𝐖xz+𝐇tβˆ’1𝐖hz+𝐛z)𝐇~t=tanh⁑(𝐗t𝐖xh+(𝐑tβŠ™π‡tβˆ’1)𝐖hh+𝐛h)𝐇t=𝐙tβŠ™π‡tβˆ’1+(1βˆ’π™t)βŠ™π‡~t

Gated Linear Unit

Gated Linear Units (GLUs) adapt the gating mechanism for use in feedforward neural networks, often within transformer-based architectures. They are defined as:GLU(a,b)=aβŠ™Οƒ(b)where a,b are the first and second inputs, respectively. Οƒ represents the sigmoid activation function.

Replacing Οƒ with other activation functions leads to variants of GLU:ReGLU(a,b)=aβŠ™ReLU(b)GEGLU(a,b)=aβŠ™GELU(b)SwiGLU(a,b,Ξ²)=aβŠ™SwishΞ²(b)where ReLU, GELU, and Swish are different activation functions, applied componentwise.

Adding weights and biases, this results in the following functions:GLU⁑(x,W,V,b,c)=Οƒ(xW+b)βŠ™(xV+c)Bilinear⁑(x,W,V,b,c)=(xW+b)βŠ™(xV+c)ReGLU⁑(x,W,V,b,c)=max⁑(0,xW+b)βŠ™(xV+c)GEGLU⁑(x,W,V,b,c)=GELU⁑(xW+b)βŠ™(xV+c)SwiGLU⁑(x,W,V,b,c,Ξ²)=SwishΞ²(xW+b)βŠ™(xV+c)These functions are often used in transformers, in the feedforward modules following an attention block: FFNGLU(x,W,V,b,c,Wβ€²,bβ€²)=(Οƒ(xW+b)βŠ™(xV+c))Wβ€²+bβ€²FFNBilinear(x,W,V,b,c,Wβ€²,bβ€²)=((xW+b)βŠ™(xV+c))Wβ€²+bβ€²FFNReGLU(x,W,V,b,c,Wβ€²,bβ€²)=(max⁑(0,xW+b)βŠ™(xV+c))Wβ€²+bβ€²FFNGEGLU(x,W,V,b,c,Wβ€²,bβ€²)=(GELU⁑(xW+b)βŠ™(xV+c))Wβ€²+bβ€²FFNSwiGLU(x,W,V,b,c,Ξ²,Wβ€²,bβ€²)=(SwishΞ²(xW+b)βŠ™(xV+c))Wβ€²+bβ€²Gated MLP (gMLP) is a module used in vision models obtained by linearly gating a set of vectors. Specifically, consider a vision model at a certain layer. It consists of a 2-dimensional array of vectors x1,…,xn, y1,…,yn, which are vertically stacked in two matrices X,Y. The vectors are linearly mixed, and added a bias, as WX+b, where W is nΓ—n in shape. Then it is used to gate Y as (WX+b)βŠ™Y.

Gated Attention Unit (GAU) is a module used in Transformers obtained by merging an attention mechanism and a linear gating feedforward module:((xW+b)βŠ™Attention⁑(x,x))Wβ€²+bβ€²

Other architectures

Gating mechanism is used in highway networks, which were designed by unrolling an LSTM.

Channel gating uses a gate to control the flow of information through different channels inside a convolutional neural network (CNN).

See also