Understanding LSTM Architecture and Computations (Step-by-Step Guide)

Sections

What Is an LSTM?
The Gates in an LSTM
Candidate Cell State
Updating the Cell State and Hidden State
Key Takeaways

What Is an LSTM?

Long Short-Term Memory (LSTM) is a special type of recurrent neural network (RNN). Unlike a basic RNN, an LSTM can remember information for longer periods. This makes it powerful for tasks like text generation, machine translation, and sentiment analysis.

At its core, an LSTM cell works with:

• Cell state (C): acts as the memory of the network.

• Hidden state (h): stores recent output information.

• Input (x): the current data point at time t.

• Output (y): prediction at time t.

In this blog, I will show you in more detail the architecture of the LSTM and the computations that are made inside a single cell. As you might recall from earlier, in a typical LSTM, you have a cell state, a hidden state, input x, and output y. You can think of the cell state as the memory of your network and gets modified using information from the input and the previous hidden state.

The Gates in an LSTM

The magic of LSTMs lies in three gates that control how information flows:

1. Forget Gate

• Decides what information from the previous cell state should be discarded.

• Uses a sigmoid activation to output values between 0 and 1.

• 0 → forget everything, 1 → keep everything.

2. Input Gate

• Selects which new information to add to the cell state.

• Also powered by a sigmoid function.

3. Output Gate

• Controls what information from the cell state becomes the hidden state and output.

• Ensures only the relevant part of the memory is passed along.

The gates in an LSTM decide which information is kept or added and select the output at every step. You will typically have three gates, your forget gate used to decide which information is kept and which is discarded from the previous cell state. The input gate that selects the relevant information from the input in previous hidden state and the output gate, that determines which information from the cell state is used as output and stored in the hidden state. Sigmoid activation functions with different trainable parameters are applied to the input and previous states for the three gates. The use of the sigmoid ensures that the values for the gates are between 0 and 1. In practice, the values encountered in the vectors corresponding to each gate are very close to either zero or one. With a value of zero, the gate is closed, so information doesn't get through. While a value of one will let information flow freely.

Candidate Cell State

In addition to the gates, there’s a candidate cell state (denoted as C~).

• It transforms the input and previous hidden state using a tanh activation.

• Values are squeezed between -1 and 1.

• The candidate cell state updates the existing memory through the input gate.

Another significant computation made inside an LSTM is the candidate cell state. To get that, you have to transform the information from the previous hidden states and the current inputs. A hyperbolic tangent activation function is typically used in different implementations. It shrinks the information from the previous hidden states and the current inputs to be between negative 1 and 1. This nonlinear transformation has been used in the past because it improves training performance.

Updating the Cell State and Hidden State

1. Update Cell State (C):

   • Forget old information using the forget gate.

   • Add new information from the candidate cell state via the input gate.

2. Update Hidden State (h):

• Apply tanh to the new cell state.

• Pass it through the output gate to decide what becomes the hidden state and output.

However, you could try other activations and test their behavior. With the forget and input gate and the candidate cell state, you can update the cell state. To get the new cell state, you add the information from the candidate cell state that passes through the input gates to the cell state information that passes through the forget gate. Finally, you can compute the new hidden state used to produce output at a given step. To get the new hidden state, you pass transformed information from the new cell state through the output gate. Note that here the new cell state first passes through a hyperbolic tangent activation. Nevertheless, some LSTM architectures ignore this information and pass the new cell state directly through the output gate. 

Key Takeaways

• LSTMs use three gates (forget, input, output) to control information flow.

• The cell state works as memory, updated at every step.

• Sigmoid functions decide what passes through, while tanh scales values.

• LSTMs are widely used in NLP tasks like text classification, sentiment analysis, and machine translation.

Let recap everything I just covered. An LSTM has three gates to decide what information is passed along the network. The forget gate decides what to keep. The input gate decides what to add from the previous hidden step and current input and the output gate determines the next hidden state and the output at a given step. You now understand the intuition behind LSTMs. In the coding exercise for this week, you'll implement LSTMs on your own. In the next video, I will show you how to use LSTMs to solve a real NLP task. Let's go to the next video.