Short answer
Computer vision is the branch of artificial intelligence that enables computers to interpret and understand images and video: recognising objects, reading text, tracking movement and measuring the world. It powers phone face unlock, self-driving cars, medical image analysis, factory inspection and photo search. Modern computer vision relies on deep learning, first convolutional neural networks (CNNs) and now increasingly vision transformers, trained on millions of labelled images and run on GPUs and edge AI chips.
What computers "see"
To a computer, an image is just a grid of numbers: each pixel has values for red, green and blue. A 12-megapixel photo is 36 million numbers. Computer vision is about turning those numbers into meaning: "this is a cat", "there is a pedestrian 20 metres ahead", "this weld is cracked".
Core computer vision tasks
| Task | Question it answers | Example |
|---|---|---|
| Image classification | What is in this image? | Labelling photos as "dog" or "cat" |
| Object detection | What objects are where? (boxes) | Detecting cars and people in traffic video |
| Semantic segmentation | Which pixels belong to which class? | Marking road, sky and pavement |
| Instance segmentation | Which pixels belong to each individual object? | Separating each person in a crowd |
| OCR | What text is in the image? | Reading receipts, number plates, documents |
| Pose estimation | Where are the body joints? | Fitness apps, motion capture |
| Tracking | Where does each object move over time? | Sports analytics, security |
| Depth and 3D | How far away is everything? | Robotics, AR, autonomous driving |
| Image generation | Create a new image | Diffusion models |
How computer vision works
Convolutional neural networks (CNNs)
A CNN slides small filters across an image to detect simple patterns such as edges and corners in early layers, then combines them into textures, parts (eyes, wheels) and whole objects in deeper layers. The breakthrough came in 2012 when AlexNet, a CNN trained on GPUs, won the ImageNet challenge by a wide margin. ResNet (2015) made very deep networks trainable and became a standard backbone.
Vision transformers (ViT)
In 2020, researchers showed that the transformer architecture from language AI works for images too: split the image into patches, treat each patch like a token and use attention to relate them. Vision transformers scale very well with data and are the basis of many modern vision and multimodal models.
Foundation models for vision
Instead of training a new model for every task, large pre-trained models can be reused:
- CLIP learns to match images with text, enabling search like "photos of red cars at night".
- Segment Anything (SAM) from Meta can cut out almost any object from a click or prompt.
- YOLO models provide fast real-time object detection on modest hardware.
- Vision language models answer questions about images in plain language.
The training pipeline
1. Collect and label images (boxes, masks or captions). 2. Train or fine-tune a model on that data. 3. Evaluate on images it has never seen. 4. Deploy in the cloud or on edge devices, often with quantization to make it smaller and faster.
Real-world uses
- Smartphones: face unlock, portrait mode, photo search, live translation of signs.
- Automotive: driver assistance, lane keeping, self-driving perception.
- Healthcare: helping radiologists spot findings in X-rays, CT scans and pathology slides.
- Manufacturing: detecting defects on production lines.
- Retail: checkout-free stores, shelf monitoring, visual product search.
- Agriculture: crop health monitoring from drones and satellites.
- Security: intrusion detection and access control, which raises important privacy questions.
- Documents: OCR for invoices, IDs and forms.
- Robotics: grasping objects and navigating spaces.
Challenges
- Data and labelling: good training data is expensive to collect and annotate.
- Edge cases: rare situations (unusual weather, odd objects) cause errors, which is critical for self-driving.
- Bias: models can perform worse for groups under-represented in training data.
- Adversarial examples: small, deliberate changes to images can fool models.
- Privacy: facial recognition and surveillance are tightly regulated in many places, including under the EU AI Act.
Hardware for computer vision
- Training uses GPUs with plenty of VRAM; vision models are generally smaller than LLMs, so a single consumer GPU can train or fine-tune many of them.
- Real-time inference on cameras, robots and cars uses edge AI hardware such as NVIDIA Jetson modules and NPUs in phones and AI PCs.
- Large-scale video analytics in the cloud uses data center GPUs, which also include hardware video decoders.
Tensor cores speed up the convolutions and matrix multiplications that dominate vision workloads.
Frequently asked questions
Is computer vision the same as image processing?
No. Image processing changes images (sharpening, resizing, filtering). Computer vision extracts meaning from them. Vision systems often use image processing as a first step.
What programming language is used for computer vision?
Python is the most common, with libraries such as OpenCV, PyTorch, torchvision and Ultralytics YOLO.
How accurate is computer vision?
On standard benchmarks, top models match or exceed human accuracy for many recognition tasks, but real-world accuracy depends on lighting, camera quality and how closely conditions match the training data.
Is computer vision part of generative AI?
Traditional computer vision analyses images; generative AI creates them. The fields overlap heavily and share architectures. See What is generative AI?.
Sources and further reading
- Krizhevsky et al., ImageNet Classification with Deep Convolutional Neural Networks (2012)
- He et al., Deep Residual Learning for Image Recognition (2015)
- Dosovitskiy et al., An Image is Worth 16x16 Words (2020)
- Kirillov et al., Segment Anything (2023)




Leave a Reply