← Back to Blog

A Brief History of AI

· 5 min read ·
AI Machine Learning LLM History Neural Networks

While I was working on a project about LoRA and RAG I went deeper into how AI — or more specifically LLMs — actually work and discovered some crazy things. This led me to want to know how all of these things came to be in the first place, and so going even deeper I found this:

This all begins in 1957 when Frank Rosenblatt introduced the Perceptron, the first trainable neural network. This was a single layer neural network and because of this it could only learn the four basic gates and not more complex things such as the XOR gate. I went much deeper into the implementation of a perceptron and it is really easy — it’s just math. Here’s a simple implementation of a perceptron in C++: perceptron.cpp. There will be a separate post coming tomorrow explaining this in detail.

To solve the XOR problem, researchers realized they needed to stack layers. If you pass data through an intermediate layer (a hidden layer), the network can distort and warp the data space, turning a non-linear problem into a linear one. But another problem arose — they could not calculate the error in the middle layer as it was not linked to the output.

In comes backpropagation, popularized by Geoffrey Hinton, David Rumelhart, and Ronald Williams in 1986. By using the Chain Rule from calculus, they proved you could calculate how much any arbitrary weight in a deep network contributed to the final error. Now errors could flow backwards through the network.

Then in 1989 Yann LeCun published a landmark paper demonstrating the first successful training of a CNN using backpropagation. In 1998 he introduced LeNet-5, which was a much more comprehensive model. This was significantly inspired by the Neocognitron developed by Kunihiko Fukushima in 1980, which predates backpropagation itself.

Despite backpropagation, networks remained shallow (3 to 5 layers) throughout the 90s and 2000s because of the Vanishing Gradient Problem. Early networks used the Sigmoid activation function, which squashes numbers between 0 and 1. The derivative of Sigmoid maxes out at 0.25. If you chain 10 layers together, multiplying 0.25 repeatedly makes the gradient shrink to almost zero before it reaches the early layers. This kept models small — LeNet-5 had around 60,000 parameters, which sounds like a lot until you compare it to what came later.

In 2012 Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered AlexNet into the ImageNet competition, crushing traditional computer vision systems. AlexNet didn’t succeed because of entirely new math; it succeeded because of engineering:

  • ReLU Activation: They replaced Sigmoid with ReLU (Rectified Linear Unit): f(x) = max(0, x). Its derivative for positive numbers is exactly 1.0, completely eliminating vanishing gradients.
  • CUDA Parallelization: They wrote custom CUDA code to run the matrix multiplications on an NVIDIA GeForce GTX 580 GPU instead of a CPU, reducing training times from weeks to days.

AlexNet had 60 million parameters across 8 layers — a thousand times more than LeNet-5. This was the moment the field realized that scale mattered. As networks grew deeper, accuracy began to degrade because deep matrix multiplications became unstable. In 2015 Kaiming He introduced ResNets (Residual Networks), which added Skip Connections to solve this. ResNet-152 pushed to 152 layers and 60 million parameters while actually being more stable than shallower networks.

Around the same time, researchers worked on sequence processing. RNNs processed text word-by-word, passing a “hidden state” memory vector from one step to the next. This created a massive engineering ceiling:

  1. No Parallelization: You cannot compute the gradient or forward pass for the 50th word until you have sequentially calculated the first 49 words. This meant they could not leverage the massive parallel computing power of modern GPUs.
  2. Memory Fade: LSTMs (Long Short-Term Memory, introduced by Hochreiter & Schmidhuber in 1997) added gated memory cells to retain information over longer sequences, but even then the hidden state vector acted as a compression bottleneck. By the time a long sentence reached token 100, the mathematical signal from token 1 was severely degraded or completely overwritten.

In 2017 Google researchers published a paper known as “Attention is All You Need” which went on to become the core of modern LLMs — or AI as we know it. This tossed recurrence completely out the window. They replaced the sequential steps with a fully parallelized matrix operation: Self-Attention. Because the entire sequence could now be processed in parallel, you could finally throw massive compute at the problem. The original Transformer had 65 million parameters. GPT-2 in 2019 had 1.5 billion. GPT-3 in 2020 had 175 billion. Today’s frontier models are estimated to be in the trillions. The architecture hadn’t changed that dramatically — the scale had. Now almost all modern LLMs are based on the core concept introduced in this paper.

This was a brief summary of how modern AI came to be. More detailed posts about things stated here to follow.