Skip to main content
Engineering4 min read

Why Voice Type turned off the Core ML encoder

whisper.cpp ships a Core ML encoder path for Apple Silicon. Voice Type force-disables it, because a Core ML encoder wants a fixed input shape and live dictation produces batches of varying length.

Key takeaways

answer-first
  • A compiled Core ML encoder bundle expects a fixed input shape.
  • Live dictation produces batches of varying length, because the batcher refuses to cut a phrase in half.
  • Voice Type therefore disables the Core ML path at context creation and runs Metal, which accepts variable-length audio.
  • Removing the path also removed a 163 MB to 1.17 GB download per model.

whisper.cpp ships an optional Core ML encoder path for Apple Silicon. It is the feature you would expect a Mac dictation app to turn on. Voice Type force-disables it and runs the Metal backend instead.

The reason is a shape mismatch, and it comes from the way live dictation batches audio.

TL;DR

  • A compiled Core ML encoder bundle expects a fixed input shape.
  • Live dictation produces batches of varying length, because the batcher refuses to cut a phrase in half.
  • Voice Type therefore disables the Core ML path at context creation and runs Metal, which accepts variable-length audio.
  • Removing the path also removed a 163 MB to 1.17 GB download per model.

Why the batches vary in length

Voice Type does not send a fixed timer slice to the recognizer. A streaming voice activity detector marks speech regions, and the batcher accumulates whole regions. It will not split one because a timer fired.

The rules that decide when a batch goes to the model are all conditional on speech, not on the clock:

  • The live batch target is 29 seconds, clamped to a range of 15 to 29 seconds.
  • A batch shorter than 5 seconds is never flushed by a pause. It waits and the next phrase joins it.
  • A pause flush needs at least 400 ms of silence after a closed speech region.
  • If adding the next whole region would cross the target, the current batch flushes first and the whole region starts the next one.
  • When you release the hotkey, whatever remains flushes regardless of length.

The consequence is that batch length is a distribution, not a constant. A batch can be 5 seconds or 31. That is the correct behaviour for dictation, because the alternative is cutting a word in half at a window boundary.

What that costs you on Core ML

A Core ML model is compiled ahead of time into an .mlmodelc bundle with its input dimensions baked in. Feed it audio of a length it was not compiled for and you either pad to the fixed shape, which means paying for encoder work on silence, or you keep several compiled variants around, which means several bundles on disk.

Voice Type takes neither option. The Core ML encoder path is force-disabled where the whisper context is created, with the comment naming exactly this: fixed-shape input constraints. The backend policy then selects Metal by default, and falls back to CPU only in the simulator, when a user default disables the GPU, or when safe mode has latched after a crash.

Metal takes whatever length the batcher produces.

What the removal freed up

Core ML support was not just disabled, it was removed. Older builds installed an encoder bundle next to each model's weights, ggml-<name>-encoder.mlmodelc, running from roughly 163 MB to 1.17 GB depending on the model. Those bundles were downloaded, unzipped, and staged per model.

Because no code path deletes them any more, the app carries a one-time migration that sweeps the orphaned encoder bundles out of the model directory on upgrade, and is careful never to touch the .bin weights beside them.

So the user-visible result of turning off Core ML is a smaller download and a smaller model directory, on top of a recognizer that does not care how long your sentence was.

Diagram of where Voice Type runs each stage. Audio capture and the Silero voice activity detector run on the CPU. The whisper.cpp recognizer runs on the Metal GPU, drawn as the main path. A third branch, Core ML on the Apple Neural Engine, is drawn dashed and disabled, labelled with the reason: a compiled Core ML encoder requires a fixed input shape and live dictation produces batches of varying length. A fallback arrow returns the recognizer to the CPU in safe mode.

The dashed branch is the path the app deliberately does not take. Source: whisper.cpp.swift/LibWhisper.swift.

What Metal actually runs

Metal is Apple's low-level GPU API, and in this app it is doing two separate jobs.

The whisper context runs on the Metal backend, so the encoder and decoder execute on the GPU. Separately, the per-batch audio conditioning runs as its own signal processing stage before the model sees anything: DC offset removal, the filters and loudness control, then resampling to the 16 kHz the model expects.

Both read from unified memory, so audio buffers are not copied across a bus between the capture path and the model.

What we are not claiming

We do not publish a finalization time here. We do not currently have a benchmark script we could hand you alongside the number, and a latency figure without a way to reproduce it is marketing.

What is structural, and true without a benchmark: the work left when you release the hotkey is bounded by the batch target, not by how long you spoke. A two-minute dictation and a thirty-minute dictation end with the same amount left to finish.

To time it on your own machine, dictate the same sentence ten times and count the gap between releasing the hotkey and the text landing. That number is the one that matters, and it is yours.

Trade-offs

Model size against accuracy. Smaller models finalize faster and miss more words. Voice Type ships several sizes, from 27 MB to 550 MB, so the trade-off is yours to set.

The larger models suit an Apple Silicon Mac with unified memory to spare. The smaller ones suit an 8 GB machine, where a larger model spends its time paging rather than decoding.

Related: Technology overview · Voice activity detection · Latency demo

Related articles

Engineering
48 kHz to 16 kHz: what band-limited resampling protects, and where it happens

Whisper-style models expect 16 kHz and most Mac hardware runs at 48 kHz. Decimating without a low-pass filter folds everything above 8 kHz back into the speech band. What band-limited conversion does, and why Voice Type asks for 16 kHz at capture instead.

Engineering
RNNoise or Silero: choosing a voice activity detector for dictation

RNNoise suppresses noise and reports speech probability as a by-product. Silero detects speech and leaves the audio alone. Why Voice Type gates with Silero, with the thresholds it actually ships.