Edge AI refers to running artificial intelligence workloads, most often model inference, directly on local devices, such as smartphones, cameras, cars, and embedded sensors, rather than sending data to a remote cloud data center for processing. It contrasts with cloud AI, where a device sends input to a server running a large model and receives the result back over a network.
Motivation
Edge AI is driven by several practical constraints that cloud inference does not solve well. Latency-sensitive applications, such as camera-based object detection in a self-driving car or real-time voice processing, cannot tolerate the round-trip delay of a network call. Offline operation matters for devices without reliable connectivity. Privacy is a major driver: processing data such as photos, voice, or health signals locally means raw personal data never has to leave the device, a property increasingly marketed by phone makers. Cost and bandwidth also favor the edge, since streaming continuous sensor data to the cloud for every inference is expensive at scale. The tradeoff is capability: edge devices have far less compute and memory than a data-center GPU cluster, so edge AI depends heavily on quantization, model distillation, and other size-reduction techniques to fit a capable model into a small memory and power footprint.
Hardware and techniques
Edge AI runs on specialized low-power chips: neural processing units built into modern smartphone systems-on-chip, dedicated inference accelerators such as Google's Edge TPU, a smaller relative of its data-center TPU, and increasingly capable embedded GPUs. Models deployed at the edge are typically compressed versions of larger models: quantized to lower numerical precision, such as 8-bit or 4-bit weights instead of 16- or 32-bit, pruned to remove unnecessary parameters, or produced via knowledge distillation from a larger foundation model into a smaller student network. Frameworks such as TensorFlow Lite, built on TensorFlow, and Core ML support this deployment path from a full-size model to an edge-optimized one.
Applications and outlook
Common edge AI applications include on-device voice assistants that process a wake word locally before invoking the cloud, computational photography and face unlock on smartphones, predictive maintenance sensors in industrial equipment, and perception systems in robotics and self-driving cars that cannot depend on network connectivity. As of the mid-2020s, capable small language models with a few billion parameters, made practical by aggressive quantization and distillation, extended edge AI beyond narrow perception tasks toward running general-purpose chat and reasoning models directly on phones and laptops, part of a broader industry push toward on-device generative AI framed by some as a privacy- and cost-driven counterweight to cloud-hosted frontier models.