What is edge AI?
Edge AI means running AI models directly on devices or on-site servers, such as cameras, sensors, phones, industrial PCs or smart displays, instead of sending data to a cloud data centre for processing. Data is analysed close to where it is created, so systems can respond faster, keep working when the network is down, and keep sensitive data on site.
Two terms are often mixed up. Edge computing is the processing infrastructure located near the data source, the "where". Edge AI is the AI application running on that infrastructure, the "what". A security camera is an edge device; when it uses computer vision to recognise people or objects on the device itself, that is edge AI.
Edge AI does not replace the cloud. In most real systems, models are trained on compute-rich infrastructure and then exported to a format suitable for devices; the cloud still handles training, reporting and distributing updates.
Edge AI vs cloud AI vs hybrid: comparison table
The table compares the three deployment options on the criteria that usually decide the choice. We do not quote specific latency numbers because results depend on your network, hardware and model; measure on your real system.
| Criterion | Edge AI (on site) | Cloud AI | Hybrid |
|---|---|---|---|
| Latency | Does not depend on the internet round trip; suits decisions that need an immediate response | Adds network round-trip time | Urgent decisions on site, the rest sent to the cloud |
| Connectivity | Can run offline | Needs a stable connection | Runs locally when offline, syncs when the connection returns |
| Privacy | Raw data can stay on site; only results are sent | Data is sent to the provider's infrastructure | Only aggregated or de-identified data is sent |
| Cost drivers | Buying and replacing devices, installation, maintenance at each site | Compute and data transfer fees by usage | Both, but less data to transfer |
| Model size | Limited by device memory and power; usually small or compressed models | Can use large models | Small model on site, large model in the cloud for hard tasks |
| Updates | Updates must reach many devices, with rollback | Updated centrally in one place | Cloud manages versions and pushes them to devices |
| Operations | Needs an accountable person at each site; debugging may require being there | Central operations team | Requires a clear design of what runs where |
Typical edge AI use cases
- Manufacturing: analysing vibration and temperature data from sensors at the machine to detect anomalies and support predictive maintenance.
- On-site computer vision: inspecting defects on a production line, counting people or detecting objects from cameras without streaming all video to the cloud.
- Retail and buildings: smart displays, kiosks and presence sensors that process data locally; see smart displays and IoT in business operations.
- Mobile devices: features that run on the phone where coverage is weak, or when users do not want content sent elsewhere.
- Handheld medical devices and remote sites: analysis at the point of care when connectivity is poor (sector regulations apply).
Hardware classes for edge AI
You do not need to pick a specific product at the start. Begin with the hardware class that fits the task, power supply and installation environment:
- Microcontrollers: very low power, run very small models for sensors or simple audio; the LiteRT documentation says integer quantization produces smaller, faster models that suit low-power devices such as microcontrollers.
- Embedded computers, single-board computers and industrial gateways: run a full operating system, suit moderate computer vision, and are often deployed with containers.
- Dedicated AI accelerators (NPUs, edge TPUs, embedded GPUs): for vision tasks or larger models at reasonable power draw.
- Phones, tablets and PCs with NPUs: use hardware your users already have.
- On-premise edge servers: serve many cameras or users at one site and can run larger models.
Getting models onto devices: quantization and small models
Edge devices have less memory and power than servers, so models need to be made compact. The two main routes are compressing a model (quantization, pruning) and choosing a model designed to be small from the start.
Quantization
Quantization represents weights (and sometimes activations) with fewer bits, for example from 32-bit floating point to 8-bit integers. According to Google's LiteRT documentation, post-training quantization can reduce model size while improving CPU and accelerator latency, with little degradation in accuracy; dynamic range quantization specifically achieves about a 4x size reduction. In the documentation's example, a ResNet-v2-101 model shrinks from 171 MB to 43 MB with the same top-1 accuracy.
The documentation also warns that accuracy can drop, particularly for smaller networks, so always re-check the quantized model on your own data; if the drop is too large, consider quantization-aware training.
Small language models on devices
For language models, the trend is toward small models that fit on a phone. Microsoft's Phi-3 technical report, for example, describes phi-3-mini as a 3.8-billion-parameter model small enough to be deployed on a phone, and the report's PDF states it can be quantized to 4 bits to occupy about 1.8 GB of memory. These are the authors' published figures; quality for Vietnamese must be measured by you. See how in choosing an LLM for Vietnamese.
Formats and tools
A common workflow: train on powerful infrastructure, export to a device format such as LiteRT (formerly TensorFlow Lite), ONNX or OpenVINO, then optimise for the target hardware.
Edge AI and privacy under Law 91/2025/QH15
Vietnam's Law on Personal Data Protection No. 91/2025/QH15 took effect on 1 January 2026. Cameras, microphones and sensors often capture personal data, so an edge architecture can help meet some of the law's principles, although it does not make a business compliant by itself.
Article 3(2) requires collecting and processing personal data only within a specific, clear scope and purpose. Processing on site and sending only results (for example a people count or an alert) instead of raw video limits how much personal data is transmitted and stored.
Article 20(1)(c) treats using a platform outside Vietnam to process personal data collected in Vietnam as a cross-border transfer, which triggers the impact assessment dossier obligation in clause 2. If personal data is processed entirely on devices or servers in Vietnam, less data leaves the country; each system still needs its own assessment.
Article 30(3) requires systems using AI and cloud computing to include appropriate security, authentication and access control; this applies to edge devices too. Article 31 has specific rules on biometric and location data, which matter directly for face-recognition cameras.
- Send only aggregated or de-identified data to the cloud when the task allows.
- Encrypt data stored on devices and the sync channel; manage keys and role-based access.
- Define retention and deletion rules for each device.
- See also data governance basics and Vietnam's Personal Data Protection Law 2025.
Checklist: should you use edge AI?
The more questions you answer "yes", the more edge AI is worth considering. If most are "no", the cloud is usually simpler and easier to improve.
- Must decisions be made almost immediately, and is the measured network latency unacceptable?
- Must the system keep running without internet?
- Should raw data (video, audio, personal data) not, or must it not, leave the site?
- Is raw data too large to stream to the cloud continuously at a reasonable cost?
- Is the model small enough, or can it be compressed while still reaching the accuracy you need on real data?
- Is someone accountable for devices at each site, with an update and rollback process?
- Have you counted full lifecycle cost: devices, installation, maintenance, replacement, physical security?
Start with one site
Pilot at one location with clear success criteria (accuracy, measured latency, uptime while offline), fix what does not work, and only then scale. When scaling, design sync carefully: what data goes to the cloud, how often, and what happens when sync fails. EYC's connected technology team can help design such a pilot; for infrastructure choices, see choosing a cloud architecture.
Frequently asked questions
What is edge AI, in short?
Edge AI means running AI models directly on devices or on-site servers, close to where data is created, instead of sending data to the cloud for processing. This lets the system respond faster, keep working offline and keep sensitive data on site.
How is edge AI different from edge computing?
Edge computing is the processing infrastructure near the data source. Edge AI is the AI application running on it, for example a camera that recognises objects on the device itself.
Is edge AI cheaper than the cloud?
Not necessarily. Edge reduces cloud compute and transfer costs but adds device, installation, maintenance and replacement costs at each site. Compare total lifecycle cost for your actual volume.
Can large language models run on a device?
Yes, small and compressed ones. The Phi-3 technical report, for example, describes the 3.8-billion-parameter phi-3-mini as small enough to run on a phone. Quality for your task and language, especially Vietnamese, needs separate testing.
Does on-device processing help with Vietnam's personal data law?
It can help, because it reduces how much personal data is transmitted and stored, and can narrow cross-border transfers under Article 20 of Law 91/2025/QH15. Edge devices still need security, access control and retention rules, and a specific legal assessment is still required.