Short answer
Edge AI means running AI models directly on a local device, such as a smartphone, laptop, security camera, car, drone, robot or factory sensor, instead of sending data to a cloud data center. Processing data where it is created gives faster responses, better privacy, lower bandwidth costs and offline operation. Edge AI runs on efficient chips like the NPUs in phones and AI PCs, NVIDIA Jetson modules and small GPUs, usually with compact, quantized models.
Edge AI vs cloud AI
| Edge AI | Cloud AI | |
|---|---|---|
| Where it runs | On the device or a nearby server | In remote data centers |
| Latency | Milliseconds, no network round trip | Depends on connection |
| Privacy | Data can stay on the device | Data is sent to a provider |
| Works offline | Yes | No |
| Model size | Small to medium | Up to the largest frontier models |
| Cost model | Hardware cost up front, low running cost | Pay per use or per GPU-hour |
| Power | Strict limits (often under 1 W to ~100 W) | Hundreds of kilowatts per rack |
Many products use both: a small on-device model handles quick, private tasks and hands harder requests to a large cloud model.
Where you already use edge AI
- Smartphones: face unlock, photo enhancement, live transcription, on-device assistants and features like Apple Intelligence and Gemini Nano.
- PCs: Copilot+ PCs with NPUs of at least 40 TOPS run features such as live captions and background effects. See What is an AI PC?.
- Cars: driver assistance and self-driving systems must react instantly, so perception runs on board.
- Smart cameras: detecting people, vehicles or packages without streaming all video to the cloud.
- Factories: inspecting products on a production line in real time.
- Robots and drones: navigation and object recognition with no network delay.
- Healthcare devices: wearables that detect irregular heart rhythms on the device.
- Retail: shelf monitoring and checkout systems.
The hardware behind edge AI
| Hardware | Typical AI performance | Used in |
|---|---|---|
| Phone NPUs (Apple Neural Engine, Qualcomm Hexagon, Google Tensor) | Tens of TOPS | Smartphones, tablets |
| PC NPUs (Intel, AMD, Qualcomm) | 40+ TOPS in Copilot+ PCs | Laptops |
| NVIDIA Jetson Orin | Up to 275 TOPS (AGX Orin) | Robots, drones, cameras, industrial systems |
| NVIDIA Jetson Thor | Data-center-class Blackwell GPU in a module | Humanoid robots and advanced robotics |
| Small GPUs and mini PCs | Varies | Edge servers, retail, local LLMs |
| Microcontroller AI (TinyML) | Very low | Sensors, wearables, appliances |
TOPS (trillions of operations per second) is the usual headline figure for edge chips; see TOPS vs TFLOPS. Real performance also depends on memory bandwidth and software support. NPUs are compared with GPUs and TPUs in NPU vs GPU vs TPU.
How models are made small enough for the edge
- Quantization: storing weights in 8 or 4 bits instead of 16 or 32. See LLM quantization.
- Distillation: training a small model to imitate a large one. See What is knowledge distillation?.
- Pruning: removing unnecessary weights.
- Efficient architectures: models designed for mobile, such as MobileNet and YOLO variants for vision and small language models for text.
- Runtimes: tools like TensorRT, ONNX Runtime, Core ML, LiteRT (formerly TensorFlow Lite) and llama.cpp optimise models for specific chips.
Benefits of edge AI
1. Speed: decisions in milliseconds, essential for cars, robots and AR. 2. Privacy: sensitive data such as faces, voices and health readings can stay on the device. 3. Reliability: keeps working without an internet connection. 4. Lower bandwidth: only results, not raw video, need to be sent. 5. Cost: avoids paying for cloud compute on every request. 6. Energy: small models on efficient chips can use far less energy per task. See how much energy AI uses.
Challenges
- Limited compute and memory: devices cannot run the largest models.
- Power and heat: batteries and passive cooling limit sustained performance.
- Fragmented hardware: each chip has its own tools and supported operations.
- Updates and security: deploying and securing models on thousands of devices is complex.
Edge AI and local LLMs
Running a chatbot on your own laptop or desktop is also a form of edge AI. Small and mid-sized models now run well on consumer GPUs and Apple Silicon. See how to run an LLM locally, how much VRAM you need and the best GPU for local LLMs.
Frequently asked questions
Is edge AI the same as on-device AI?
Mostly yes. On-device AI is edge AI running on the end device itself. Edge AI can also include small servers placed close to where data is created, such as in a store or factory.
Do I need an NPU for edge AI?
No, but NPUs are the most power-efficient option on phones and laptops. GPUs and CPUs can also run edge models.
What is TinyML?
TinyML is machine learning on microcontrollers that use milliwatts of power, such as keyword detection ("Hey Siri") or vibration monitoring in machines.
Will edge AI replace cloud AI?
No. The biggest, most capable models will stay in the cloud. Edge and cloud AI complement each other.
Sources and further reading
- NVIDIA, Jetson AGX Orin
- Microsoft, Copilot+ PCs
- Howard et al., MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017)
- Google, LiteRT documentation




Leave a Reply