How Neural Networks Learned to Remember — RNNs, LSTMs, and GRUs
It’s an afternoon of a beautiful summer day. You just learnt about backpropagation, gradients and artificial neural networks and you think you’ve unlocked some forbidden treasure of knowledge granted to only few those who seek.
Welcome to the construct. Everything you thought was magic is just weighted sums.
You’re finally thinking neural networks in terms of functions, approximators, matrices and what not.
But just then, you’re hit with BIG WORDS like RNNs, CNNs and now Transformers ( wasn’t it a movie ? )
So, why were ANNs not enough? Why the need for these complex architectures?
ANNs perform very well with independent, non-related inputs.
Basically an input -> output mapping.
For instance,your vitals with health diagnosis, your credit scores and income with loan approval recommendation etc
Turns out, most real life data are a lot more complex, and a lot more interrelated to each other.
The selfie you took with your friends — pixels related spatially.
The caption you put with it — text order matters.
The fix to your selfie is a whole different domain in itself -
‘CNNs’. Perhaps, I’ll write about it someday later. Lets keep our focus on text as of now.
ANN will treat these two “captions” the same:
Sentence 1: Smiling because life is good, not because I have to
Sentence 2: Smiling because I have to, not because life is good
You see the problem now.
How do we solve this?
Behold…here comes the magic formula — “Recurrency”
The fix is almost embarrassingly simple once you see it. If the problem is that the network forgets what came before, then give it a memory and make it read the sentence the way you do: one word at a time, left to right, carrying what you've read so far in your head.
That "what you've read so far in your head" is the whole idea. We call it the hidden state, h. At every step the network looks at the current word and its own memory from the previous step, mashes them together, and produces a new memory.
In one line of math:
h_t = tanh(W_x · x_t + W_h · h_{t-1} + b)
Read it slowly, because everything about RNNs lives in this equation:
x_tis the current word (its embedding).h_{t-1}is the memory from the previous step — everything the network has seen up to now, squished into a fixed-size vector.W_xandW_hare weight matrices.W_xdecides how much the new word matters,W_hdecides how much the old memory matters.tanhsquashes it all into a tidy range so the numbers don't explode.
The output h_t becomes the input memory for the next step. It's a loop. That's literally why it's called recurrent — the network feeds its own output back into itself.
Here's the part people gloss over: the weights are the same at every step. There isn't a fresh W_x for word 1 and a different one for word 2. It's one little cell, reused over and over as you slide across the sentence. The network isn't memorizing positions; it's learning one rule for "how to update memory given a new word," and applying it everywhere. That weight sharing is what lets it handle sentences of any length with a fixed number of parameters.
If you draw it out across time, you get the classic "unrolled" picture — the same cell, copy-pasted at each timestep, passing memory rightward:
Now run our two captions through it. Word by word, the hidden state after "Smiling because life is good" is in a genuinely different place than the hidden state after "Smiling because I have to." By the time the network reaches the end, those two memories have diverged, and the final h reflects which sentence it actually read. Order survived. That's the win.
In code, the naive version is just a for loop over time:
import torch
import torch.nn as nn
# The mechanics, by hand — so you see there's no magic
hidden_size, input_size = 8, 4
W_x = nn.Linear(input_size, hidden_size) # word -> memory
W_h = nn.Linear(hidden_size, hidden_size) # memory -> memory
h = torch.zeros(hidden_size) # blank memory to start
sequence = torch.randn(5, input_size) # a 5-word "sentence"
for x_t in sequence: # walk left to right
h = torch.tanh(W_x(x_t) + W_h(h)) # the recurrence, one step
# h now summarizes the whole sequence
In real life you don't hand-roll the loop. PyTorch ships the whole thing as one module, and it runs the loop in optimized C for you:
rnn = nn.RNN(input_size=4, hidden_size=8, batch_first=True)
seq = torch.randn(1, 5, 4) # (batch, time, features)
outputs, h_n = rnn(seq)
# outputs: the hidden state at *every* step -> (1, 5, 8)
# h_n: just the final memory -> (1, 1, 8)
That's an RNN. Memory in a loop. For a while in the 2010s this was the answer for anything that looked like a sequence — text, speech, sensor streams, stock prices.
And then you try to use it on a real sentence and it falls on its face.
Where the vanilla RNN quietly breaks
Here's a sentence:
The cat, which had already eaten a giant bowl of food earlier and was clearly in no mood for more, was full.
To pick "was" over "were," the network has to remember that the subject was "cat" (singular) — a decision it made something like fifteen words ago, through a whole clause of distractions. In theory the hidden state carries that. In practice, vanilla RNNs are terrible at it.
The reason is gradients. When you train an RNN, you unroll it across all those timesteps and backpropagate through every one of them — there's even a name for it, backpropagation through time (BPTT). And here's the catch: to push an error signal from "was" all the way back to "cat," the gradient has to travel back through every step in between, getting multiplied by roughly the same W_h at each hop.
Multiply a number smaller than 1 by itself fifteen times and it collapses toward zero. Multiply a number bigger than 1 by itself fifteen times and it explodes to infinity. Those are the two failure modes, and they have names too: vanishing and exploding gradients. tanh makes it worse — it saturates, flattening to a near-zero slope whenever the input gets large, which drains the gradient even faster.
Exploding gradients you can mostly duct-tape with gradient clipping. Vanishing gradients are the real killer. When the gradient vanishes, the early words simply stop sending any learning signal to the later ones. The network physically cannot learn the long-range dependency, no matter how long you train it. The memory has a short half-life, and it's much shorter than you'd hope — a handful of steps, not the whole paragraph.
In practice the effective memory is embarrassingly short. A plain RNN struggles to hold a dependency much beyond 10 to 20 steps before the signal washes out — and that's on a good day. It'll happily complete a short fragment and then spit out total nonsense the moment the thing it needs to remember sits more than a handful of words back. It isn't undertrained. It's structurally forgetful.
So the question stopped being "how do I train this better" and became "how do I change the architecture so memory survives the trip."
LSTM — giving the network a memory it can actually protect
The fix, from Hochreiter and Schmidhuber back in 1997 (yes, this idea is older than most people building with it), is the Long Short-Term Memory network. The name is the whole thesis: keep short-term memory around for the long term.
The trick is to add a second track. Alongside the hidden state, an LSTM carries a cell state, C — think of it as a conveyor belt running straight through the sequence. Information can ride the belt across many steps almost untouched. Crucially, the main thing the belt does is add and remove information with cheap, mostly-linear operations, instead of completely rewriting memory through a tanh at every step the way the vanilla RNN does. That additive path is a gradient highway: gradients can flow back across dozens of steps without getting multiplied into oblivion. That, in one sentence, is why LSTMs beat the vanishing-gradient problem.
What controls what goes on and off the belt? Gates. A gate is just a sigmoid layer that outputs numbers between 0 and 1 — a soft on/off knob — multiplied against a vector. 0 means "block this completely," 1 means "let it all through." An LSTM has three:
Forget gate — looks at the new word and the old memory and decides what to wipe from the belt. ("New subject showed up? Drop the old subject's number.")
Input gate — decides what new information from the current step is worth writing onto the belt.
Output gate — decides what part of the belt to expose as this step's hidden state.
The equations, once, so you've seen them:
f_t = σ(W_f · [h_{t-1}, x_t] + b_f) # forget: what to erase
i_t = σ(W_i · [h_{t-1}, x_t] + b_i) # input: how much new stuff to write
g_t = tanh(W_g · [h_{t-1}, x_t] + b_g) # the candidate new info
C_t = f_t * C_{t-1} + i_t * g_t # update the belt: erase, then add
o_t = σ(W_o · [h_{t-1}, x_t] + b_o) # output: what to reveal
h_t = o_t * tanh(C_t) # this step's exposed memory
Don't memorize them. Just notice the shape of the one that matters, the C_t line: the old cell state gets scaled by the forget gate and then added to the new candidate. Scale and add. No repeated matrix multiply mangling the memory at every step. That's the conveyor belt, and that's the whole reason "cat … was" now works across fifteen words.
In PyTorch it's a one-word change from before — nn.RNN becomes nn.LSTM, and now you get back two memory tracks instead of one:
lstm = nn.LSTM(input_size=4, hidden_size=8, batch_first=True)
seq = torch.randn(1, 5, 4)
outputs, (h_n, c_n) = lstm(seq) # note: hidden state AND cell state
# outputs: hidden state at every step -> (1, 5, 8)
# h_n: final hidden state, c_n: final cell state (the belt)
This is the architecture that made sequence models actually useful. Google Translate ran on stacked LSTMs. Early speech recognition, handwriting, the first wave of decent autocomplete — LSTMs under the hood. Swap a plain RNN for an LSTM on a task that actually needs memory and the difference is night and day: suddenly it can hold a thread across a whole sentence instead of dropping it after a few words.
GRU — the lighter cousin that's almost as good
LSTMs work, but they're a bit of a Rube Goldberg machine — two memory states, three gates, a pile of weight matrices. In 2014 Cho et al. asked the obvious question: can we get most of the benefit with less machinery?
That's the Gated Recurrent Unit. It makes two cuts:
It throws out the separate cell state. There's just the hidden state again — the conveyor belt and the exposed memory are merged into one.
It drops to two gates instead of three:
Update gate — does the job of the LSTM's forget and input gates at once: how much of the old memory to keep versus how much new stuff to write.
Reset gate — decides how much of the past to ignore when computing the new candidate memory.
z_t = σ(W_z · [h_{t-1}, x_t]) # update gate
r_t = σ(W_r · [h_{t-1}, x_t]) # reset gate
n_t = tanh(W_n · [r_t * h_{t-1}, x_t]) # candidate memory
h_t = (1 - z_t) * h_{t-1} + z_t * n_t # keep old vs. take new
Look at that last line — it's the same "scale and add" trick that saved the LSTM, just with one knob (z_t) sliding between "keep the old memory" and "use the new one." Same gradient highway, fewer parts.
gru = nn.GRU(input_size=4, hidden_size=8, batch_first=True)
seq = torch.randn(1, 5, 4)
outputs, h_n = gru(seq) # back to a single memory track, like the RNN
So which one do you actually reach for? Here's my take, and I'm not going to hedge it: start with a GRU. It has roughly 25% fewer parameters than an LSTM for the same hidden size (three gate-ish matrices instead of four), it trains faster, it needs less data to fit, and on the large majority of tasks the accuracy difference is in the noise. The famous LSTM-vs-GRU bake-offs basically came back "it depends, and the gap is small." When the gap is small, take the cheaper, faster model.
I'd reach for the LSTM specifically when the sequences are long and you genuinely need the extra control of a dedicated, separately-gated memory cell — very long documents, tasks where you've actually measured the GRU topping out. Otherwise the LSTM's extra gate is paying rent it doesn't earn. Don't agonize over it; try the GRU, and only upgrade if your eval tells you to.
So… does any of this still matter?
I'll be honest about the elephant: you are probably not going to ship a new product on an LSTM in 2026. Transformers ate this entire field.
And the reason why is buried in the very first equation in this post. Look at the recurrence again: h_t depends on h_{t-1}. To compute step 100, you must have already computed step 99, which needed step 98, all the way down. The whole thing is inherently sequential. You cannot parallelize a sentence across your GPU — you're stuck processing it one token at a time, and that's death when you're trying to train on billions of tokens. LSTMs didn't lose on quality first. They lost on throughput.
Attention threw that constraint out. It looks at every token's relationship to every other token in one parallel shot — no waiting for the previous step, no gradient crawling back through fifty hops. Same job, no handbrake. That's the whole reason it won.
But "obsolete" is too strong, and the people declaring RNNs fully dead are overshooting. There are real corners where recurrence still wins:
Streaming and real-time inference. An RNN carries a single fixed-size memory vector and updates it per token, in constant time and constant memory. A vanilla Transformer's attention cost grows quadratically with sequence length. For something like a live voice agent chewing through an unbounded audio stream — exactly the kind of latency-sensitive thing you'd build from Kathmandu without a rack of spare GPUs lying around — that constant per-step cost is genuinely attractive.
Tiny edge models. A small GRU running on a microcontroller for keyword spotting or sensor anomaly detection is lean in a way a Transformer struggles to match.
It's not even fully settled. The recent crop of state-space models (Mamba and friends) is, in spirit, the recurrence idea coming back around with the parallelism problem solved. The conveyor belt didn't die. It evolved.
And here's the real reason to sit with all of this even if you only ever build with the shiny stuff: the concepts are the foundation, not the implementation. Hidden state as compressed memory. Gates as learned, soft control over what to keep and what to forget. The whole fight to keep gradients alive across distance. None of those ideas went anywhere — they got reshuffled into attention and everything after it. Skip them and the modern stuff is a black box. Learn them and you understand why the modern stuff is shaped the way it is.
Which is exactly where we're going next. The follow-up post takes apart the thing that ate this entire field — attention and Transformers — and why ripping out the recurrence loop turned out to be the whole trick. Wasn't a movie after all.
So: what's the longest dependency you've watched a model completely fail to learn — and did you fix it with a better architecture, or just throw more data at it and hope?
