The Accelerator That Quietly Says No
A phone is four or five processors wearing one name. The neural engine is the fastest and the fussiest, and when it refuses part of your model it does not raise an error — it hands the work back to the CPU and lets you believe you are accelerated.
Part 5 of a ten-part series on the engineering problems that define edge AI.
Ask where a model runs on a phone and the natural answer is “on the phone.” That answer conceals the thing you most need to know. The chip in a modern handset is not a processor; it is a collection of quite different processors sharing a package, and which one your model lands on can change its speed and energy consumption by more than an order of magnitude.
Worse, that choice is often not made explicitly by you. It is made by a runtime, at load time, silently, and it is frequently made differently than you assume.
Four processors, four personalities
It helps to have a concrete mental model of what is actually on the die.
The CPU is the generalist. It will run absolutely anything, correctly, today. It is also the least efficient way to do large-scale tensor arithmetic — the most energy per unit of useful work by a wide margin. It is where things end up when nothing better will take them.
The GPU handles wide parallel arithmetic well and is far more efficient than the CPU for that shape of work. It is more flexible than the specialized units and less efficient than them, which makes it a common and reasonable middle ground.
The DSP was designed for continuous signal processing at very low power. It excels at fixed-point arithmetic over streams and is often the right home for the always-on tier of a cascade — the gate that must run constantly without noticeably touching the battery.
The NPU, variously called a neural engine or neural accelerator, is purpose-built silicon for the specific operations that dominate neural networks. For work it accepts, it is dramatically faster and more efficient than everything else on the chip — frequently by ten or twenty times against the CPU.
That phrase, for work it accepts, is where all the difficulty lives.
Fast and narrow
The reason the NPU is so efficient is the reason it is so restrictive: it does not implement a general computer. It implements a fixed set of operations, at fixed numeric precisions, over tensors of predictable shape, wired together in hardware. Everything it does not implement, it does not do.
The constraints tend to fall into a few recurring categories.
Operator coverage. The accelerator supports a specific list of operations. Standard convolutions, standard activations, ordinary matrix multiplies — these are almost always present. Anything less common may not be: unusual dilation patterns, certain normalization layers, custom operations, newer attention variants, and nearly always the operator you thought was completely ordinary.
Numeric precision. Many accelerators are built for 8-bit integer arithmetic and support floating point partially or not at all. A model that has not been quantized may be ineligible on those grounds alone, which links this article directly to Part 3 — quantization is often not an optimization but an admission requirement.
Shape rigidity. Accelerators want to know tensor dimensions when the graph is compiled, not when it runs. A model whose input length varies with the data — very common in audio and text — may be rejected outright or forced into a padded fixed shape, and the padding is real work you are paying for.
Memory limits. Dedicated on-chip memory is small. Models or intermediate tensors exceeding it may be partitioned in ways that cost more in data movement than the acceleration returns.
The silent fallback
Here is the specific behavior that costs beginners the most time, and it is worth being blunt about how it presents.
When a runtime encounters an operator the accelerator cannot handle, it does not fail. It does not warn. In most frameworks, by default, it does not even log. It partitions the graph — runs the supported portion on the accelerator, hands the unsupported operator to the CPU, and hands the result back to continue.
Your model runs. It produces correct output. Every API call reports success. And it may be running mostly on the CPU while you believe it is accelerated.
The performance consequence is worse than merely losing the acceleration for that one operator, because every boundary between compute units is a data transfer. Tensors must be moved, and often converted between numeric formats and memory layouts on the way. A single unsupported operator sitting in the middle of a model forces two of these round trips. A handful of unsupported operators scattered through the graph can produce a dozen partitions, and a model split into a dozen pieces frequently runs slower than if you had simply told it to use the CPU for everything — because you are now paying full CPU cost for part of the work plus a great deal of shuttling that pure-CPU execution never needed.
This produces one of the more disorienting results in edge deployment: enabling the accelerator makes the model slower. It is not a paradox. It is graph partitioning, and it is entirely diagnosable once you know to look.
Look at the partition report
Which is the practical heart of this article. Every mature runtime can tell you where each operator was placed. Almost nobody asks.
The tooling differs by platform — a compute-unit assignment report, a delegate log, a graph visualizer that colors nodes by execution provider — but the information is always available, and the first thing to establish when deploying a model to an accelerator is not how fast it runs. It is:
- How many partitions did the graph get split into? One is ideal. A handful is workable. A dozen is a problem.
- Which specific operators fell back, and why?
- What fraction of total computation is actually running on the accelerator?
Answer those three and you will usually know exactly what to do next, and the answer will be much better targeted than any amount of general optimization. Skip them and you are optimizing in the dark, which in practice means optimizing the part that was already fast.
Write in the accelerator’s vocabulary
Once you know which operators are falling back, the fix is usually to express the same computation differently — the mathematics is unchanged, the operators are ones the hardware knows.
The general moves recur across platforms. An exotic activation function can be replaced with a common one, often with negligible accuracy cost and a large placement benefit. An unusual normalization can sometimes be folded into an adjacent operation at export time. A custom operator can be decomposed into a sequence of standard ones. Dynamic shapes can be replaced by a small set of fixed shapes — bucketing inputs into a few sizes and padding to the nearest — which trades a little wasted computation for full acceleration and usually wins comfortably.
There is a broader lesson here that applies well beyond any specific chip. Architecture choices made during training determine deployability, and the decision is being made whether or not anyone is thinking about it. A model designed with the target accelerator’s operator set in mind can be twenty times faster than a mathematically similar model that used one unsupported layer in a hot path. Choosing that layer was a five-minute decision during research. Discovering its cost happens months later, during deployment, when changing it means retraining.
The habit that prevents this is cheap: export the model to the target runtime and check the partition report early — while the architecture is still soft. Not after training converges. Not the week before ship. A ten-minute check during the first week of a project routinely saves a retraining cycle at the end of it.
The rest of the surprises
A few more behaviors are worth knowing in advance, because each one produces measurements that look like model problems and are not.
Compilation is not free. Many runtimes compile the graph for the target unit on first load, which can take seconds. This is a first-run cost, and it must be excluded from steady-state latency measurements and accounted for in cold-start user experience. Caching the compiled artifact is usually available and usually forgotten.
The accelerator is shared. It is a single resource, and other applications — and the operating system itself — use it too. Your model may queue. This is a significant source of the latency tail discussed in Part 2, and it does not appear in single-app benchmarking.
Numerical results differ between units. The same model on CPU and NPU will not produce bit-identical output; the accelerator’s internal precision and accumulation order differ. Usually irrelevant. Occasionally not — a threshold comparison right at a decision boundary can flip. Validate accuracy on the unit you will actually ship, not on the one you developed against.
Efficiency cores are not performance cores. Even “the CPU” is several different processors with different speeds. Work scheduled onto an efficiency core runs several times slower, and the scheduler moves threads based on system-wide conditions you do not control. This is a routine cause of inexplicable latency spikes and a subject Part 9 returns to.
The stance to adopt
The instinct carried over from server-side work is that hardware is an implementation detail — write the model, let the runtime handle placement. On a device that instinct is expensive, because the runtime’s default behavior is to silently do something reasonable rather than to tell you it could not do what you wanted.
The stance that works is closer to the one you would take toward a compiler you did not fully trust: assume nothing about placement, read the report, and verify that the thing you believe is happening is the thing that is happening. The accelerator will not tell you it said no. It will simply hand the work back, and let the benchmark you did not look closely enough at tell you everything is fine.
Next in this series: why the fanciest index is usually the wrong answer on a device.