Skip to content

What Is Computer Vision? How AI Sees and Understands Images

Computer vision is the field of AI that teaches computers to understand images and video. Core tasks like classification, object detection and segmentation, how CNNs and vision transformers work, real-world uses, and the hardware behind it.

4 min read

What Is Computer Vision? — cover image from TensorCore.si

Short answer

Computer vision is the branch of artificial intelligence that enables computers to interpret and understand images and video: recognising objects, reading text, tracking movement and measuring the world. It powers phone face unlock, self-driving cars, medical image analysis, factory inspection and photo search. Modern computer vision relies on deep learning, first convolutional neural networks (CNNs) and now increasingly vision transformers, trained on millions of labelled images and run on GPUs and edge AI chips.

What computers "see"

To a computer, an image is just a grid of numbers: each pixel has values for red, green and blue. A 12-megapixel photo is 36 million numbers. Computer vision is about turning those numbers into meaning: "this is a cat", "there is a pedestrian 20 metres ahead", "this weld is cracked".

Core computer vision tasks

TaskQuestion it answersExample
Image classificationWhat is in this image?Labelling photos as "dog" or "cat"
Object detectionWhat objects are where? (boxes)Detecting cars and people in traffic video
Semantic segmentationWhich pixels belong to which class?Marking road, sky and pavement
Instance segmentationWhich pixels belong to each individual object?Separating each person in a crowd
OCRWhat text is in the image?Reading receipts, number plates, documents
Pose estimationWhere are the body joints?Fitness apps, motion capture
TrackingWhere does each object move over time?Sports analytics, security
Depth and 3DHow far away is everything?Robotics, AR, autonomous driving
Image generationCreate a new imageDiffusion models

How computer vision works

Convolutional neural networks (CNNs)

A CNN slides small filters across an image to detect simple patterns such as edges and corners in early layers, then combines them into textures, parts (eyes, wheels) and whole objects in deeper layers. The breakthrough came in 2012 when AlexNet, a CNN trained on GPUs, won the ImageNet challenge by a wide margin. ResNet (2015) made very deep networks trainable and became a standard backbone.

Vision transformers (ViT)

In 2020, researchers showed that the transformer architecture from language AI works for images too: split the image into patches, treat each patch like a token and use attention to relate them. Vision transformers scale very well with data and are the basis of many modern vision and multimodal models.

Foundation models for vision

Instead of training a new model for every task, large pre-trained models can be reused:

  • CLIP learns to match images with text, enabling search like "photos of red cars at night".
  • Segment Anything (SAM) from Meta can cut out almost any object from a click or prompt.
  • YOLO models provide fast real-time object detection on modest hardware.
  • Vision language models answer questions about images in plain language.

The training pipeline

1. Collect and label images (boxes, masks or captions). 2. Train or fine-tune a model on that data. 3. Evaluate on images it has never seen. 4. Deploy in the cloud or on edge devices, often with quantization to make it smaller and faster.

Real-world uses

  • Smartphones: face unlock, portrait mode, photo search, live translation of signs.
  • Automotive: driver assistance, lane keeping, self-driving perception.
  • Healthcare: helping radiologists spot findings in X-rays, CT scans and pathology slides.
  • Manufacturing: detecting defects on production lines.
  • Retail: checkout-free stores, shelf monitoring, visual product search.
  • Agriculture: crop health monitoring from drones and satellites.
  • Security: intrusion detection and access control, which raises important privacy questions.
  • Documents: OCR for invoices, IDs and forms.
  • Robotics: grasping objects and navigating spaces.

Challenges

  • Data and labelling: good training data is expensive to collect and annotate.
  • Edge cases: rare situations (unusual weather, odd objects) cause errors, which is critical for self-driving.
  • Bias: models can perform worse for groups under-represented in training data.
  • Adversarial examples: small, deliberate changes to images can fool models.
  • Privacy: facial recognition and surveillance are tightly regulated in many places, including under the EU AI Act.

Hardware for computer vision

  • Training uses GPUs with plenty of VRAM; vision models are generally smaller than LLMs, so a single consumer GPU can train or fine-tune many of them.
  • Real-time inference on cameras, robots and cars uses edge AI hardware such as NVIDIA Jetson modules and NPUs in phones and AI PCs.
  • Large-scale video analytics in the cloud uses data center GPUs, which also include hardware video decoders.

Tensor cores speed up the convolutions and matrix multiplications that dominate vision workloads.

Frequently asked questions

Is computer vision the same as image processing?

No. Image processing changes images (sharpening, resizing, filtering). Computer vision extracts meaning from them. Vision systems often use image processing as a first step.

What programming language is used for computer vision?

Python is the most common, with libraries such as OpenCV, PyTorch, torchvision and Ultralytics YOLO.

How accurate is computer vision?

On standard benchmarks, top models match or exceed human accuracy for many recognition tasks, but real-world accuracy depends on lighting, camera quality and how closely conditions match the training data.

Is computer vision part of generative AI?

Traditional computer vision analyses images; generative AI creates them. The fields overlap heavily and share architectures. See What is generative AI?.

Sources and further reading

Think you understand it now? Prove it.

Solve real Tensor Core math in the TensorCore.si Challenge and earn a certificate with your name.

Take the challenge

Leave a Reply

Your email address will not be published. Required fields are marked *