Skip to content

What Is a Neural Network? How Artificial Neural Networks Work

A neural network is a machine learning model made of layers of simple connected units that learn patterns from data. How neurons, weights and layers work, how training with backpropagation works, the main types of neural networks and why they run on GPUs.

4 min read

What Is a Neural Network? — cover image from TensorCore.si

Short answer

A neural network is a machine learning model loosely inspired by the brain. It is made of layers of simple units called neurons, connected by adjustable numbers called weights. Each neuron multiplies its inputs by weights, adds them up and applies an activation function. During training, the network makes predictions, measures its errors and adjusts the weights with backpropagation and gradient descent until it gets good at the task. Neural networks power image recognition, speech recognition and large language models, and they run on GPUs because their core operation is matrix multiplication.

The basic idea

Traditional programs follow rules a person writes. A neural network learns its own rules from examples. Show it thousands of labelled photos of cats and dogs, and it gradually adjusts its internal numbers until it can tell them apart, including in photos it has never seen.

How a neuron works

An artificial neuron does three simple things:

  • Multiply: each input is multiplied by a weight that says how important it is.
  • Add: the weighted inputs are summed, plus a number called the bias.
  • Activate: the result goes through an activation function, such as ReLU (which turns negative numbers into zero), so the network can learn non-linear patterns.

On its own a neuron is very simple. The power comes from connecting millions or billions of them.

Layers: input, hidden and output

LayerWhat it doesExample (recognising handwritten digits)
Input layerReceives the raw data784 pixel values from a 28x28 image
Hidden layersFind increasingly complex patternsEdges, then curves, then shapes
Output layerGives the final answer10 scores, one for each digit 0 to 9

A network with many hidden layers is called a deep neural network, which is where the term deep learning comes from. See AI vs machine learning vs deep learning.

How neural networks learn

Training repeats four steps millions of times:

  • Forward pass: feed an example through the network to get a prediction.
  • Loss: measure how wrong the prediction is with a loss function.
  • Backpropagation: work backwards through the network to calculate how much each weight contributed to the error (the gradient).
  • Gradient descent: nudge every weight slightly in the direction that reduces the error.

The size of each nudge is set by the learning rate. Backpropagation was popularised in a famous 1986 paper by David Rumelhart, Geoffrey Hinton and Ronald Williams, and it is still how almost every neural network is trained today. Training at scale uses tricks like mixed precision training to run faster.

Types of neural networks

TypeGood atExamples
Feedforward (MLP)Tabular data, simple classificationCredit scoring
Convolutional (CNN)Images and videoPhoto tagging, medical imaging
Recurrent (RNN, LSTM)Sequences (now mostly replaced by transformers)Older speech and translation systems
TransformerLanguage, and increasingly images, audio and videoChatGPT, Claude, Gemini
Diffusion modelsGenerating images and videoImage generators

Transformers are the architecture behind today's large language models; see what a transformer is.

A short history

  • 1958: Frank Rosenblatt builds the perceptron, an early single-layer neural network.
  • 1986: backpropagation makes it practical to train multi-layer networks.
  • 2012: AlexNet, a deep CNN trained on two NVIDIA GPUs, wins the ImageNet competition by a large margin and starts the deep learning boom.
  • 2017: the transformer architecture is introduced.
  • 2020s: large language models with billions to trillions of parameters.

Why neural networks need GPUs

Almost all the work in a neural network is multiplying large matrices of weights and activations. GPUs have thousands of cores that do these multiplications in parallel, and modern GPUs add tensor cores built specifically for this maths. That is why AI progress has tracked GPU progress so closely. See GPU vs CPU for AI and how a GPU works.

Frequently asked questions

Is a neural network the same as the human brain?

No. It is loosely inspired by neurons, but artificial neurons are simple mathematical functions and networks learn very differently from brains.

What is the difference between a neural network and deep learning?

Deep learning means using neural networks with many layers. All deep learning uses neural networks, but a very small network is not usually called deep learning.

How many neurons does a neural network have?

Anything from a handful to billions. Large language models are usually described by their number of parameters (weights), which can be in the hundreds of billions.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *