SpecEnvoy

EN·中文

‹ Back to the post

Under the hood: the model stack

Published 7 June 2026 · the technical companion to "What's inside StarlingType"

Exactly which models StarlingType runs, what they can and can't do, who built them, and an honest "is this state-of-the-art?" verdict for mid-2026.

StarlingType's pipeline is wake word → speech-to-text → inject at cursor. There is no LLM and no text-to-speech (those belong to a sibling project). So there are really just three model files, plus the runtime that executes them.

The stack at a glance

Every piece is Apache-2.0 and runs fully on-device via ONNX — which is what lets the same models port from Mac to phones without re-engineering the ML.

1. The runtime: Sherpa-ONNX (Next-gen Kaldi)

What it is: not a model — the engine that loads and runs the .onnx files. It wraps Microsoft's ONNX Runtime and provides ready-made speech components (keyword spotter, offline/streaming recognizers, VAD) with bindings for Python, Swift, Kotlin, and more.

Who made it: the Next-gen Kaldi team (GitHub k2-fsa), led by Daniel Povey — creator of the original Kaldi toolkit, the most influential pre-deep-learning speech framework. The modern stack is k2, icefall (training recipes), lhotse (data), and sherpa / sherpa-onnx (deployment).

2. The wake-word models (KWS = keyword spotting)

StarlingType runs two keyword spotters in parallel — one Chinese, one English — so either language can wake it.

Architecture: both are Zipformer-based transducer keyword spotters (~3.3M parameters, ~13 MB each). Zipformer is the Next-gen Kaldi team's ASR encoder ("Zipformer: A faster and better encoder for ASR," ICLR 2024) — a genuinely SOTA-class encoder that has topped LibriSpeech-style benchmarks. The KWS models use the icefall open-vocabulary recipe: you supply keyword token sequences plus detection thresholds, and the model fires on a match.

Who made the training corpora (distinct from who trained the models): WenetSpeech is a 10,000+ hour multi-domain Mandarin corpus from the WeNet open-source community (Binbin Zhang et al., ICASSP 2022); GigaSpeech is a 10,000 hour English corpus from SpeechColab (Guoguo Chen et al., Interspeech 2021). The Next-gen Kaldi team trained the wake-word models on these.

What KWS can do: listen continuously at low cost and fire on predefined wake phrases (e.g. 嘿小听 / "Hey Type").

What KWS cannot do: it is not a general transcriber — only the configured phrases. And detection quality is empirical, not guaranteed by token validity: a phrase can have every token in the vocabulary and still detect poorly (which is why StarlingType's wake words were chosen by real-world testing). Rhotic-R tokens collapse for non-rhotic accents; novel brand words detect worse than common words. These are inherent limits of small KWS models, managed by phrase selection rather than a "better model."

3. The speech-to-text model: SenseVoice-Small

What it is: SenseVoice-Small, the packaged zh-en-ja-ko-yue variant (~280 MB) — a non-autoregressive (CTC-style) multilingual speech-understanding model. It transcribes an utterance in essentially one forward pass instead of token-by-token, which is what makes it fast and low-latency.

Who made it: Alibaba's FunAudioLLM / Tongyi speech team (the lineage of Alibaba DAMO Academy's speech lab; FunASR / ModelScope ecosystem). Released July 2024, Apache 2.0. Trained on 400,000+ hours; the full family advertises 50+ languages (the on-device build here is the focused 5-language one).

What it can do:

• ASR with automatic language ID per utterance — the core of "switch languages mid-sentence."
• Strong Chinese & Cantonese — independent comparisons find it beats Whisper on Chinese and under-resourced languages.
• Very fast — the paper reports ~10 s of audio in ~70 ms, roughly 15× faster than Whisper-Large, with a small footprint → ideal for the <500 ms latency target.
• Bonus (StarlingType uses little of this): emotion recognition, audio-event detection, inverse text normalization.

What it cannot do well:

• Not built for long-form / timestamped transcription — it's tuned for short utterances; no word-level timestamps, not designed to stream a 45-minute file the way Whisper is.
• No speaker diarization.
• English accuracy is good but not class-leading — its win is multilingual + speed + size, not English purity.
• Like all ASR: struggles with heavy noise, overlapping speakers, rare proper nouns.

"Are these SOTA?" — the honest mid-2026 picture

STT moved fast since these models shipped in 2024. The 2026 open landscape:

• English accuracy champ: NVIDIA Canary-Qwen-2.5B tops the Hugging Face Open ASR Leaderboard (~5.6% avg WER). IBM Granite Speech is another top English option.
• Speed champ: NVIDIA Parakeet (TDT/CTC) — extreme throughput for long-form batch English.
• Multilingual breadth: Whisper Large-v3 / v3-Turbo; Qwen3-ASR is the natural successor to SenseVoice.

Bottom line: StarlingType is not running the single highest-WER-on-a-leaderboard model — and it shouldn't, because those champions are English-centric, large, or batch-oriented. For the real constraint set (bilingual, on-device, private, low-latency, Apache-2.0, portable), the stack is best-in-class, from two of the most credible open speech teams on the planet.

Upgrade paths

• STT refresh: Qwen3-ASR is the obvious successor to SenseVoice from the same lineage — re-evaluate when a sherpa-onnx-friendly export exists.
• Long-form transcription (a different product need): Whisper-via-sherpa-onnx or Parakeet — SenseVoice would need VAD-chunking to fake long-form.
• Wake words: stay on Zipformer KWS; gains come from phrase selection and thresholds, not a bigger model.

What StarlingType deliberately does NOT use

• No LLM — no chat, no rewriting, no "smart" commands.
• No TTS — it only listens and types; it never speaks.
• No cloud STT — ever. A cloud fallback would violate the privacy pillar.