A camera that needs flip_camera, a thermostat that needs set_eco_mode, a headset that needs
mute_mic: these are commands nobody else defines, and no general voice assistant exposes an API to
add them. Building a custom on-device recognizer the usual way is a multi-month project. Record audio
across many speakers, train a speech model, then squeeze it inside the target chip’s power and memory
budget. This pipeline does it from typed phrases alone, with zero recordings, and ships the result as
one self-contained bundle for Qualcomm edge silicon. It is live at
intent.unified-sciences.com: type your commands, train, download
the bundle.
The result reads as three numbers, but it came from one question asked backward: what makes the usual pipeline take weeks, and can each of those costs be deleted rather than optimized? The rest of this is that derivation, in order, including the checks that had to pass and the single number this post refuses to headline.
Why cloud voice doesn’t fit an edge product
Cloud recognition is the obvious shortcut, and it fails an edge product on four axes at once, each a hard product constraint rather than a preference:
- Privacy. Raw audio leaves the device.
- Latency. A network round trip on every command.
- Availability. No network, no voice.
- Cost. A per-request bill that scales with usage.
For wearables, appliances, cars, and factory-floor tools, those four are the line between a feature that ships and one that stays a demo. So the target is fixed: a custom recognizer, built from nothing but typed phrases, running entirely on-device. Everything below is a search for how to delete the “weeks of work plus recordings” baseline instead of merely speeding it up.
Explore: where the cost actually is
Decompose “train a custom recognizer” and the cost concentrates in exactly two places.
- Collecting speaker audio. The microphone-and-many-speakers bottleneck: recruit speakers, record, clean, label.
- Fitting a speech model. Hours of GPU training per customer, and a large model to ship per customer.
Each cost looks mandatory for a different reason, and each reason turns out to be an assumption rather than a law. The fitting cost looks mandatory because the speech model is assumed to have to learn each customer’s commands. It does not have to. Turning audio into a feature vector is a customer-independent problem, so one shared encoder can solve it for everyone and never train again. The collecting cost looks mandatory because a classifier needs labeled speech. It does not need recorded speech, though. Text-to-speech plus enough augmentation manufactures the labeled set from the typed phrases alone. Two costs, two assumptions, two ways out.
The load-bearing realization is that both are removable, and removable independently. Neither removal depends on the other one working. That independence is the whole reason a weeks-long project can collapse to a one-minute build: the two savings multiply instead of trading off against each other.
Why attack the decomposition instead of optimizing the pipeline?
Optimizing the conventional pipeline (a smaller architecture, faster GPUs, cheaper microphones) shaves a constant factor off each stage but keeps both stages. The stages themselves are the cost. Removing a stage beats speeding it up, and the two stages here share no structure, so they can be attacked with two unrelated techniques at the same time. That is the difference between a 2× change and a 100× change: one optimizes the baseline, the other deletes it.
Remove the fitting cost: freeze the encoder, train almost nothing
Turning audio into a meaningful feature vector is a general, customer-independent problem. It has nothing to do with any one customer’s command set. So solve it once, with a shared pre-trained speech encoder whose weights are held constant, and train nothing there. The only genuinely customer-specific mapping is feature vector to this customer’s labels, and that mapping is a tiny classifier.
That single decision is the entire speed and size story. “Training” stops being fine-tuning a speech model (hours on a GPU, a fresh large model per customer) and becomes fitting a small classifier on cached feature vectors (seconds, and one shared encoder file for everyone).
The decision keeps paying past the fitting cost, which is what makes it more than a shortcut. A frozen encoder guarantees one thing the rest of the system compounds on. Its output for a given clip is fixed forever. Because of that one guarantee, two problems that arrive later (an entire extra augmentation layer, and the risk that quantizing the encoder to int8 moves the features out from under the head) collapse to almost nothing. The freeze is not one saving. It is the root of a short cascade, and the rest of this post follows that cascade link by link.
Why 220 full-batch steps take seconds, not hours
Because the encoder is frozen, every training clip’s embedding is computed once and cached. From then
on the optimizer sees a fixed matrix of vectors, not audio. A two-layer network over a few hundred
cached vectors is a trivial full-batch problem, so 220 steps is a matter of seconds rather than the
hours a speech-model fine-tune would cost. There is no data loader, no forward pass through a large
acoustic model per step, and nothing to stream from disk. The step count is fixed in code
(for _ in range(220) in backend/trainer.py), not tuned per customer.
What the '~15 s' clock in the portal actually times
The portal frames head-only training as roughly 15 seconds and the full build (list commands, synthesize, embed, train, package) as about a minute. Read those as descriptive UX copy, not isolated benchmarks. The trainer’s own timer wraps synthesis, embedding, and training together, so the 220 steps are never timed on their own in any log. The defensible claim is structural, that a full-batch fit over a few hundred cached vectors is seconds-class work, not a stopwatch reading on the training loop in isolation.
The frozen-versus-trained split is the thesis in one picture: exactly one box in the chain learns anything customer-specific.
- shared + constant
- trained per client
Remove the collecting cost: synthesize every clip
No customer audio is ever recorded. Every training clip is synthesized from the phrases the customer types, then augmented hard so a two-layer head trained on synthetic speech still generalizes to real speakers. The synthesis runs in three stacked layers, and the freeze decision quietly adds a fourth for free.
| Layer | What varies | Range / setting | Applied |
|---|---|---|---|
| Text | synonyms, carriers, verb bridge, EDA edits | synonym map + 10 prefixes / 5 suffixes | per phrase |
| Voice | 5 TTS accents, rate, pitch | rate −12% to +12% (5 steps) · pitch ±15 Hz (3 steps) | per clip |
| Audio | speed, reverb, gain, noise | 0.9–1.1× p0.7 · IR 30–120 ms p0.4 · gain −6 to +3 dB always · noise 5–25 dB SNR p0.85 | per clip |
| Feature | Gaussian copies, mixup | std 0.1× per-dim · mixup Beta(0.4, 0.4) | on cached embeddings, free |
The highlighted feature-space layer costs zero extra synthesis. It exists only because the encoder is frozen and embeddings are cached once.
Text. A command-domain synonym map (take/capture/get, photo/picture/pic/shot), carrier templates (10 prefixes such as “can you”, 5 suffixes such as “ please“), and a capture-verb bridge that swaps verbs only within a media family. That last rule is why the generator produces “record a video” and “shoot a video” but never “film a photo”. No LLM sits in this loop; the map is hand-built and domain-scoped, which is exactly why it cannot drift into nonsense.
Why a hand-built synonym map instead of an LLM paraphraser
An LLM paraphraser is the tempting default, and it fails a specific way here: it produces fluent cross-family nonsense like “film a photo” or “screenshot the video”, which then get spoken, embedded, and taught to the head as valid examples of a command. Every one of those is a mislabeled training point. A hand-built, domain-scoped map cannot cross a media family by construction. The capture-verb bridge encodes the one rule that matters, that verbs swap inside the photo family or inside the video family but never across, and light EDA-style edits (stopword drop, adjacent-word swap) add surface variety without semantic drift. The cost is manual curation. The payoff is that no training label is ever wrong.
Voices. Each expanded phrase is spoken by 5 TTS voices spanning US, UK, Indian, and Australian
accents (Microsoft’s edge-tts), with randomized speaking rate (−12% to +12% across 5 steps) and
pitch (±15 Hz across 3 steps).
Audio. Speed perturbation 0.9–1.1× (p=0.7), synthetic reverb from a random exponential-decay impulse response of 30–120 ms (p=0.4), gain of −6 to +3 dB (always applied), and additive white or pink noise at 5–25 dB SNR (p=0.85), simulating rooms and cheap microphones.
How the room reverb is faked without a single real recording
Real reverb needs a recorded room impulse response, which would reintroduce the microphone the whole design is built to avoid. Instead each clip is convolved with a synthetic exponential-decay impulse response of random length in the 30–120 ms range, applied with probability 0.4. It is not acoustically exact, and it does not need to be: the head only has to learn that the same command survives a smeared, echoey rendering. Paired with additive white or pink noise at 5–25 dB SNR (p=0.85) and a gain jitter of −6 to +3 dB, the synthetic clips span “quiet close-talk” to “noisy far-field” without anyone entering a room.
The compounding move: augmentation that costs nothing
Here the two removals stop being independent and start to multiply. Because the encoder is frozen, every clip’s embedding is computed once and cached. A cached embedding is just a vector, so a third kind of augmentation becomes available at zero marginal cost: perturb the vectors directly instead of re-synthesizing audio.
Two perturbations run in embedding space. Gaussian-noise copies add noise with standard deviation
equal to 0.1 times each dimension’s own embedding standard deviation, and whole-batch mixup blends
embeddings under a single shared λ drawn from Beta(0.4, 0.4) per step over a batch permutation.
Neither costs a TTS call or an encoder pass. This is the direct dividend of freezing: a decision made
to save training time turns out to change the economics of the data pipeline too.
There is one more link, and it only becomes visible in the next section. Training the head on embedding-space noise does more than add cheap variety. It teaches the head that small movements of the vector carry no meaning, and that tolerance is a general property, not a speaker-specific trick. So the same noise that lets a synthetic-trained head survive real speakers also absorbs a completely unrelated perturbation for free, namely the rounding error introduced when the encoder is quantized to int8. Follow the chain back and it starts from one job. Freezing was chosen to remove the fitting cost. The fixed embeddings that follow from it make training seconds-class, and they throw off two more dividends the choice was never made for, a free augmentation layer and a head already tolerant to the quantization step still to come. Two unearned payoffs, one mechanism.
Why embedding-space augmentation would be expensive on an unfrozen encoder
If the encoder were being fine-tuned, its weights would change every step, so any cached embedding
would be stale after the first update and every augmented example would need a fresh forward pass
through the full acoustic model. Freezing makes the embedding a constant, which makes caching correct,
which makes perturbing the cache equivalent to (and far cheaper than) perturbing the audio and
re-encoding. The per-dimension noise scale (0.1× that dimension’s std) keeps the perturbation
proportional to how much each feature actually varies, and mixup’s Beta(0.4, 0.4) favors blends near
the two endpoints, so most synthetic points still sit close to a real class rather than in the muddle
between two.
The three audio layers plus the free fourth turn a handful of typed phrases into a full training set. Strengthening the text and audio stages widened that fan-out on a 3-phrase dev check from 42 clips to 360. Read the bars below as a single smoke test of the generator’s reach, not a benchmark:
Verify: quantize once, then prove the vectors didn’t move
The encoder is quantized to int8 exactly once, offline (backend/build_encoder.py), not inside the
per-customer build. Every download therefore ships the identical 8,582,074-byte (8.58 MB) int8
encoder. Quantization creates a specific risk: it could shift the feature vectors out from under a head
that was trained on the fp32 vectors. If production feeds the head different vectors than training did,
accuracy rots silently, and nothing in the customer’s build would catch it.
So the move is gated, not assumed. The int8 encoder’s output must sit above 0.99 cosine similarity to the fp32 reference before anything ships. Passing that gate is the proof that the feature vectors the head trained on are the ones it sees in production. Without the gate, freeze-then-quantize is a hope. With it, it is a checked invariant.
Why int8 is safe here when it wrecks other models
Quantization error is dangerous when it lands on a sensitive downstream computation. Here the downstream is a two-layer linear head reading a mean-pooled embedding, and cosine similarity above 0.99 means the direction of the embedding (which is what a linear classifier keys on) is almost unchanged. There is also a second, non-obvious guard: the head was trained with Gaussian embedding noise at 0.1× per-dimension std, so it already expects vectors to wobble by more than int8 rounding moves them. The augmentation that makes the head generalize across speakers doubles as its tolerance for quantization noise. One mechanism, two payoffs.
Make unknown a real class, not a threshold
A shipped device is guaranteed to hear speech that is none of its commands: side conversation, a cough,
a different app’s wake word. The lazy handling is a confidence threshold on the N-way softmax,
reject if max prob < tau. That is a fragile proxy for “none of these”, and it fails exactly when the
out-of-vocabulary utterance happens to project confidently onto some command.
So unknown is a genuine trained class. The head’s output width is N+1; the +1 is a real softmax
class fit against 12 generic synthesized negatives (“what time is it”, “tell me a joke”, “never mind”,
and nine more). A device that hears out-of-vocabulary speech gets a positive unknown prediction it
can act on, ignoring or re-prompting, instead of misfiring a real command.
This is the same instinct that gates the int8 encoder above. Both replace a runtime hope (a threshold that might catch every out-of-vocabulary phrase, a quantization that might leave the features untouched) with something the build trains or checks outright, so nothing load-bearing rests on luck.
Why a trained negative beats a confidence threshold
A threshold assumes out-of-vocabulary audio always lands in a low-confidence region of the softmax. It
does not. An unrelated phrase can align with a command’s embedding well enough to clear any fixed
tau, producing a confident wrong answer, while a threshold set high enough to catch that case starts
rejecting valid commands too. Training an explicit unknown class instead carves out a region of
embedding space that means “not a command”, so rejection becomes a positive prediction with its own
learned boundary rather than a gap in someone else’s. The real limit is the negative set: 12
English-only phrases is a starting distribution, and real open-world speech is far more varied.
Evaluate on held-out synthetic voices
The easiest way to lie to yourself here is to let training phrasings or voices leak into the test. So
evaluation is kept deliberately clean. It uses each command’s original, unexpanded phrasing at neutral
prosody (rate +0%, pitch +0 Hz), spoken by 2 voices never used in training (en-US-AriaNeural,
en-GB-RyanNeural). Train and eval voices are quarantined, so the number measures generalization to
unseen speakers rather than memorization of the ones the head was fit on.
Self-improve under a real production constraint
The demo deploy runs on a CPU-only VM (e2-standard-4, no GPU), and running it there taught us exactly
where the one-minute build binds. The text-variant expansion factor runs at EXPAND_N=3 in the
deploy’s systemd unit, down from the local default of EXPAND_N=4, to hold CPU training time inside the
minute. So the binding cost is text expansion, not embedding or head training, which points at a precise
next lever. Caching synthesized audio across builds, or adding a GPU deploy tier, buys the workstation’s
EXPAND_N=4 diversity back without spending the minute. Customer-facing VM models see slightly less
text variety than a from-source workstation run today, and the path to closing that gap is already known
rather than open-ended.
What ships
The bundle is self-contained in the strict sense. The runtime depends only on onnxruntime, numpy,
scipy, and soundfile. No PyTorch, no transformers, no network calls at inference. The int8 encoder
is 8,582,074 bytes, the head is plain NumPy arrays, and the full downloadable bundle compresses to
about 5.6 MB. One shared encoder file backs every customer; only the small head and its label map
change per command set.
Where Qualcomm fits, and what's left to build
The shared encoder is exported to ONNX and quantized to int8, which is the format Qualcomm’s NPU tooling ingests, and the runtime already avoids PyTorch, so the on-device dependency surface is small. Two steps separate the current fp32 profile from the number the product will ship against. First, compile the int8 graph to the Hexagon NPU: today AI Hub’s device compilers (TFLite, QNN context-binary, and QNN-DLC) reject the quantized transformer graph, which is why the profiled build is still fp32. Second, profile on earbud-class hardware rather than a phone-class Snapdragon. The groundwork both steps depend on, the int8 export (cosine 0.99 against fp32) and the PyTorch-free runtime, is already done.
What it replaces, and by how much
Shipping this on-device buys more than privacy. A conventional custom-voice pipeline runs speech-to-text and then a small language model to parse intent: faster-whisper (base.en) into a 270M model takes about 719 ms per command on an M4 Pro. The frozen-encoder model answers the same command in a single forward pass, 16 ms for tiny and 24 ms for base on the same machine, which is 30 to 45× faster than the cascade and 14 to 21× faster than the 270M parser alone. Less compute per command is less energy per command: measured as power times active compute-time, the tiny model spends roughly an order of magnitude less than the cascade for the same result. That energy figure is a compute-time proxy on the M4, not a metered reading, and these speedups are M4 measurements, separate from the on-device Snapdragon profile above.
The architecture buys a second efficiency for free. A cloud recognizer streams raw audio off the device; this one sends only the parsed action. A two-second clip is tens of kilobytes on the wire, while the action it resolves to is a 50-byte JSON object: about 1,000× fewer bytes, and over a 400 kbps BLE link roughly 1 ms of transfer instead of about a second. On a battery-powered earbud or sensor, that radio time is often the dominant energy cost, not the inference.
What’s next
Four levers are scoped, each pointing at the next measurement rather than a missing one:
- Measure the frozen-versus-unfrozen gap directly. “Freezing costs nothing” is a design argument today, sound but not yet a head-to-head number. The next run trains a fully fine-tuned (unfrozen) encoder as the baseline and puts the two accuracies side by side.
- Put real audio in the eval set. The held-out split already proves generalization across TTS voices and phrasings unseen in training. Adding a small real-user, real-room, real-microphone set is what turns that into an end-user accuracy claim.
- Recover full text diversity on the VM. The CPU VM runs
EXPAND_N=3against the local4. Caching synthesized audio across builds, or a GPU deploy tier, buys the workstation’s variety back inside the one-minute budget. - Widen the
unknowndistribution. Twelve generic negatives seed the reject region well enough to beat a threshold today. Broadening them, and going past English-only, is the next lever for open-world coverage.
Where this goes
The architecture is the whole result, and it is the part that carries forward. Freeze one shared encoder, train only a tiny head, synthesize every clip, and gate the two risky moves (int8 drift and train/eval leakage) with checks that have to pass. That turns a multi-week, microphone-dependent project into a roughly one-minute build that fits in 8.58 MB and runs with zero network calls, and none of it is tied to one command set, so every new customer rides the same frozen encoder and the same cascade of free dividends behind it. The build is live and separate from this post at intent.unified-sciences.com, and the measurement that matters next is the int8 build profiled on earbud-class silicon, which turns the fp32 phone-class profile it already has into the latency the product will ship against.