Short answer
A neural network is a machine learning model loosely inspired by the brain. It is made of layers of simple units called neurons, connected by adjustable numbers called weights. Each neuron multiplies its inputs by weights, adds them up and applies an activation function. During training, the network makes predictions, measures its errors and adjusts the weights with backpropagation and gradient descent until it gets good at the task. Neural networks power image recognition, speech recognition and large language models, and they run on GPUs because their core operation is matrix multiplication.
The basic idea
Traditional programs follow rules a person writes. A neural network learns its own rules from examples. Show it thousands of labelled photos of cats and dogs, and it gradually adjusts its internal numbers until it can tell them apart, including in photos it has never seen.
How a neuron works
An artificial neuron does three simple things:
- Multiply: each input is multiplied by a weight that says how important it is.
- Add: the weighted inputs are summed, plus a number called the bias.
- Activate: the result goes through an activation function, such as ReLU (which turns negative numbers into zero), so the network can learn non-linear patterns.
On its own a neuron is very simple. The power comes from connecting millions or billions of them.
Layers: input, hidden and output
| Layer | What it does | Example (recognising handwritten digits) |
|---|---|---|
| Input layer | Receives the raw data | 784 pixel values from a 28x28 image |
| Hidden layers | Find increasingly complex patterns | Edges, then curves, then shapes |
| Output layer | Gives the final answer | 10 scores, one for each digit 0 to 9 |
A network with many hidden layers is called a deep neural network, which is where the term deep learning comes from. See AI vs machine learning vs deep learning.
How neural networks learn
Training repeats four steps millions of times:
- Forward pass: feed an example through the network to get a prediction.
- Loss: measure how wrong the prediction is with a loss function.
- Backpropagation: work backwards through the network to calculate how much each weight contributed to the error (the gradient).
- Gradient descent: nudge every weight slightly in the direction that reduces the error.
The size of each nudge is set by the learning rate. Backpropagation was popularised in a famous 1986 paper by David Rumelhart, Geoffrey Hinton and Ronald Williams, and it is still how almost every neural network is trained today. Training at scale uses tricks like mixed precision training to run faster.
Types of neural networks
| Type | Good at | Examples |
|---|---|---|
| Feedforward (MLP) | Tabular data, simple classification | Credit scoring |
| Convolutional (CNN) | Images and video | Photo tagging, medical imaging |
| Recurrent (RNN, LSTM) | Sequences (now mostly replaced by transformers) | Older speech and translation systems |
| Transformer | Language, and increasingly images, audio and video | ChatGPT, Claude, Gemini |
| Diffusion models | Generating images and video | Image generators |
Transformers are the architecture behind today's large language models; see what a transformer is.
A short history
- 1958: Frank Rosenblatt builds the perceptron, an early single-layer neural network.
- 1986: backpropagation makes it practical to train multi-layer networks.
- 2012: AlexNet, a deep CNN trained on two NVIDIA GPUs, wins the ImageNet competition by a large margin and starts the deep learning boom.
- 2017: the transformer architecture is introduced.
- 2020s: large language models with billions to trillions of parameters.
Why neural networks need GPUs
Almost all the work in a neural network is multiplying large matrices of weights and activations. GPUs have thousands of cores that do these multiplications in parallel, and modern GPUs add tensor cores built specifically for this maths. That is why AI progress has tracked GPU progress so closely. See GPU vs CPU for AI and how a GPU works.
Frequently asked questions
Is a neural network the same as the human brain?
No. It is loosely inspired by neurons, but artificial neurons are simple mathematical functions and networks learn very differently from brains.
What is the difference between a neural network and deep learning?
Deep learning means using neural networks with many layers. All deep learning uses neural networks, but a very small network is not usually called deep learning.
How many neurons does a neural network have?
Anything from a handful to billions. Large language models are usually described by their number of parameters (weights), which can be in the hundreds of billions.
Sources and further reading
- Rumelhart, Hinton and Williams, Learning representations by back-propagating errors (Nature, 1986)
- Krizhevsky, Sutskever and Hinton, ImageNet Classification with Deep Convolutional Neural Networks (2012)
- 3Blue1Brown, Neural networks video series




Leave a Reply