Edge AI: choosing MCU, embedded Linux or cloud
Updated 24 September 2026
Source-checked: 24 September 2026.
Choose the processing location around the product’s job, not the popularity of a model. A small sensor classifier, a camera inspection system and a conversational interface have very different resource needs. The useful question is: which decisions must the device make, under which operating conditions, and what should happen when it cannot make them?
Google’s May 2026 LiteRT-LM announcement shows how on-device generative AI is expanding across hardware backends. That makes local inference worth evaluating, but a demonstrated phone or laptop workload is not evidence that the same model fits your microcontroller. Keep platform choice tied to measurements from the intended product.
Start with a one-page decision contract
Describe the input, the decision and the permitted action. For a vibration monitor, the input might be a window of accelerometer samples; the output might be an anomaly score; the action might be logging an event for later review. Write down what happens when samples are missing or the score is uncertain. Do not silently turn a monitoring feature into a machine-protection function.
Specify a response-time budget, energy budget, memory allowance and acceptable loss of connectivity. Also identify who can update the model, how the version is recorded, and which decisions remain under conventional firmware control. These are design inputs. They should exist before comparing processor marketing figures.
When an MCU is a sensible starting point
A microcontroller unit, or MCU, is a candidate for a compact, bounded inference task close to its sensor. Check the actual model operators and runtime support before committing the PCB. Google documents a restricted operator set, a low-level C++ interface and no on-device training support for LiteRT for Microcontrollers. Those are integration constraints, not merely deployment details.
Budget the complete application. Model weights are only one consumer of memory: sensor buffers, intermediate tensors, communications, stacks and the existing firmware also need space. Keep a written allocation and a measured margin. If a model only fits after removing diagnostic logging, the apparent saving may create a support problem later.
When to evaluate embedded Linux or a hybrid
Evaluate an embedded Linux platform when the application needs a richer software stack or a workload that does not fit the MCU candidate. Treat the larger platform as a new system responsibility: boot behaviour, storage, updates, thermal limits and component support all enter the design. More compute does not remove the need for an explicit failure policy.
A hybrid can separate a local decision from a remote service. For example, retain local sensing and a defined alarm path, while a server handles fleet analysis or user explanations. Decide what crosses the network, why it is needed and how long it is retained. Local inference alone does not make the entire product private if telemetry or diagnostic uploads still transmit sensitive information.
Benchmark the pipeline, not just the model
Google’s performance guidance recommends profiling the chosen model and examining operator costs. It also highlights data-copy overhead and the resource trade-offs of multithreading. Use that approach, then extend the measurement to the complete product path.
Illustrative budget: suppose an event must produce an output within 100 ms. Allocate 20 ms to acquisition and preprocessing, 40 ms to inference and 10 ms to the decision and output path. That totals 70 ms and leaves 30 ms of budget. These are planning assumptions, not measured device performance. Instrument each stage and replace every allowance with evidence.
Record model version, runtime, clock settings, input shape, temperature conditions and concurrent activity. Test cold start as well as repeated operation. Include representative difficult inputs, not just a convenient demonstration clip. Compare the outcome quality and timing together: a faster model that makes the wrong product decision has not met the requirement.
Architecture review checklist
- The required decision and permitted action are written down.
- The deployed model’s operators are supported on the intended runtime.
- Measured memory includes buffers, activations and the surrounding application.
- Response time covers acquisition through output, with operating conditions recorded.
- Missing data, uncertain predictions and network loss have defined behaviour.
- Model updates, rollback responsibilities and version traceability are planned.
- Data leaving the product is documented separately from where inference runs.
Before choosing a board, compare two realistic architectures against the same decision contract. DVAR’s AI and machine learning integration and electronic hardware development services connect that software decision to the power, interfaces and verification needs of the physical product.