Canto: A speech model built for the real world

Canto: a speech model built for the real world Speech recognition models have become remarkably good at transcribing clean audio recorded under controlled conditions. But real dictation rarely happens under those conditions. Millions of people use Wispr Flow to message friends, write emails, code, and work through ideas at their desks, between meetings, during commutes, and in busy offices. They speak through laptop microphones, earbuds, and headsets, often with other voices, music, or traffic in the background. Today, the Wispr Advanced Interfaces Lab is introducing Canto, our latest speech model for real-time dictation. On an evaluation of real-world dictations, Canto achieved the lowest word error rate among all the models we tested. We compared Canto with models from Google, OpenAI, AssemblyAI, and Deepgram. Canto is the first model in a broader research and development program at Wispr Advanced Interfaces Lab. In this post, we share how it performs, how we trained it to handle challenging real-world conditions, and the research already shaping what comes next. “Today, the Wispr Advanced Interfaces Lab is introducing
Canto, our latest speech model for real-time dictation.” Ariya Rastrow CSO, Wispr Flow Evaluating Canto in real-world conditions To test Canto in real-world conditions, we created an evaluation set composed of 10 hours of English-language Wispr Flow dictations from more than 2,300 unique speakers, randomly sampled across applications and use cases. We were careful to enforce a strict separation between speakers represented in the train and test sets to avoid overfitting on speaker characteristics. Every sample came from a user who opted in to Wispr’s data-sharing setting, which allows their data to be used anonymously to evaluate and improve our models. More information about this setting is available in our data-controls documentation. Canto achieved the lowest Word Error Rate (WER) of the models in our comparison. WER measures word substitutions, omissions, and insertions relative to a human-transcribed reference (lower is better). Performance under the most challenging conditions The model performed well on randomly sampled data, but we were also interested in studying its performance in the most challenging situations. We built a separate, 3-hour challenge evaluation set that featured those conditions most likely to cause dictation to fail. The set includes audio affected by nearby speech, music, traffic, wind, low recording volume, and whispered or far-field speech. It also includes short dictations that give a model very little surrounding context and language to help resolve ambiguity. On the full challenge set, Canto ranked second behind Gemini 3.1 Pro, a much larger frontier-size multi-modal model that is not suitable for real-time low-latency applications. Among the real-time transcription models we evaluated, Canto achieved the lowest WER. We further examined this dataset to understand how the models behave differently. Gemini 3.1 Pro achieved the lowest WER on noisy audio. Canto tied for the lowest WER on low-volume speech and short dictations. Short dictations produced the highest error rates across the comparison. A single mistake has a larger effect on WER when an utterance contains only a few words, and the models also have less linguistic context available to resolve ambiguity. Comparing performance on public benchmarks We also evaluated Canto on three public English datasets: LibriSpeech, FLEURS, and Common Voice. Canto tied for the lowest WER on LibriSpeech. It remained competitive on FLEURS and Common Voice, although it did not lead either evaluation. These public datasets provide useful, reproducible comparisons, but they largely contain read speech. LibriSpeech is drawn from audiobooks, while FLEURS and Common Voice consist primarily of people reading prepared sentences. They capture a different distribution from everyday “in-the-wild” dictation, where people speak spontaneously, pause, revise their thoughts, and often provide very little linguistic context. The contrast helps explain why we evaluate Canto on both public datasets and real Wispr usage. Post-training Canto with Supervised Fine-Tuning and Reinforcement Learning Canto starts from a model pretrained on millions of hours of speech and text. We then train Canto in two stages. First, we show it audio paired with reference transcripts. The model learns to predict the words in each transcript, one step at a time. This stage, called supervised fine-tuning, teaches it how to perform the transcription task. Next, we train it to compare the quality of complete transcripts. For the same audio, the model generates several possible transcriptions. We score each one against a reference, then use those scores to make better transcriptions more likely in future training. This is known as reinforcement learning, or RL. The distinction is in how the model receives feedback. Supervised fine-tuning provides the expected words at each step. Reinforcement learning evaluates the completed transcription (generated by the model). That lets us train around the outcomes we care about, such as reducing recognition errors under difficult recording conditions. Our approach uses Group Relative Policy Optimization, or GRPO. The central idea is simple: compare the candidate transcripts within each group and learn from their relative scores. For each audio example, we sample several candidate transcripts from the model being trained. We call these candidates rollouts. Each receives a reward: a numerical score measuring how well it satisfies the training objective. GRPO compares each candidate’s reward with the rewards of the other candidates for the same audio. The resulting relative score, called an advantage, indicates whether that candidate performed better or worse than its group. These advantages guide the training update. Sequence-level training has a long history in speech recognition. Researchers have previously optimized ASR systems using minimum word error rate training and policy-gradient methods. GRPO was introduced more recently in DeepSeekMath, and direct applications to autoregressive speech recognition have only begun to appear in recent work on GRPO for ASR and speech-domain adaptation. “We built infrastructure that can generate and score speech rollouts at scale, allowing us to construct training environments around specific behaviors and failure modes.” For Wispr, the value of RL is not just a lower aggregate error rate. We built infrastructure that can generate and score speech rollouts at scale, allowing us to construct training environments around specific behaviors and failure modes. That system now supports experiments in contextual speech recognition, personalization, diarization, and difficult audio conditions. The research described below extends beyond the training used for the released Canto model and is already informing the next generation of our models. Learning from corrections The usage of names and vocabulary changes faster than speech models can be retrained. A word such as “Claude” might once have been an uncommon alternative to “cloud.” As its usage changes, a practical dictation model needs a way to learn the distinction. For users who have opted into data sharing, corrections can provide a valuable signal for improving recognition. But not every edit points to a transcription error. People also change formatting, rewrite sentences, or simply change their minds. A hypothetical example below illustrates this problem. The model transcribed “Claude” as “cloud,” which the person corrected. They also changed their mind and rewrote the end of the sentence. Only the first change represents a recognition error. We use signals from the audio and the edit itself to identify corrections that are most likely to represent recognition errors. These include forced-alignment confidence, which measures how well candidate words match the audio, and the location and shape of the edit. We then graft only the likely correction into the original transcript, leaving the rest unchanged. The resulting transcript becomes the reference used to score GRPO rollouts. A hypothesis receives a higher reward when it makes the intended correction. Because GRPO centers the advantages within each group, words shared by higher-reward and lower-reward hypotheses receive offsetting signals. This does not provide perfect token-level credit assignment, but it makes the strongest contrast more local to the decision we care about. Grafting turns a noisy document-level edit into a more focused sequence-level training signal while preserving the on-policy nature of GRPO. Learning to use context selectively At runtime, Canto is provided with specialized vocabulary from a person’s dictionary. This helps the model recover names and rare terms that may be difficult to identify from audio alone. Providing phrases to guide recognition is a longstanding area of contextual ASR research. The difficult part is deciding how strongly the model should trust that context. Suppose “Barry” appears in someone’s dictionary and they dictate, “I want to make a berry salad.” In poor recording conditions, both words may be acoustically plausible. An overeager model might choose “Barry” because it appears in the dictionary, even though “berry” is what the person said. To measure this behavior, we started from an SFT checkpoint and trained the model using RL while providing truthful context on 10% of training examples. We then tested the model with synthetic distractors that varied in phonetic similarity to the correct word. A distractor flip occurs when adding one of these false suggestions causes the model to adopt it. After RL training, the model became more responsive to context, including when that context was wrong. Among the most phonetically similar terms, it adopted the distractor nearly five times as often as the initial SFT model. The training had made the context more useful, but the model sometimes trusted it too readily. To separate context adoption from context discrimination, we trained three runs from the same SFT checkpoint. In always true, the supplied context contained only the correct term. In distractors + true, the correct term appeared alongside phonetically similar alternatives. In distractors only, selected examples contained plausible near-misses instead of the correct term, while the reward continued to favor the transcript supported by the audio. At every checkpoint, we ran two probes. The vertical axis measures adoption of correct context, while the horizontal axis measures how often false context changes the transcript. The ideal trajectory moves upward without moving to the right. The trajectories suggest that each mixture teaches a different relationship to context. With always true, the model can learn a simple shortcut: when a supplied term resembles the audio, use it. This improves adoption of correct terms, but also increases false-context following. With distractors + true, context becomes a selection problem. The correct answer is present, but it must be distinguished from plausible alternatives. The model is rewarded for choosing the candidate best supported by the audio, not merely for copying from the list. With distractors only, context becomes adversarial. Blindly adopting any supplied term now loses reward, so success requires resolving the conflict in favor of the audio. This pushed false-context following further left while preserving a similar level of correct-context adoption. These experiments do not yet define a final context policy, but they show that the tradeoff itself is trainable. The goal is not simply to make the model follow context, but to also teach the model when context deserves to be followed. What comes next Canto is the first model in a larger family. We are already training its successor at more than ten times Canto’s scale, with the goal of improving recognition under difficult audio conditions, improving multi-speaker recognition (for our recently launched notetaker product), making better use of contextual vocabulary, and reaching the same standard across more languages. The post-training approach described here will also let us introduce more specialized rewards and harder training environments as the models grow. One focus is a unified architecture for transcription and diarization. Meeting-transcription systems often recognize words and identify speakers in separate stages, allowing errors in one stage to compound errors in the other. A joint model can reason about what was said, who said it, and when the speaker changed at the same time. Speaker structure can also provide useful evidence in noisy settings, helping distinguish the conversation being transcribed from unrelated speech in the background. Over time, our speech models will need to do more than recognize words. They will need to use context selectively, understand the application in which someone is speaking, and connect what was said with what the person is trying to accomplish. Canto gives us the speech model, training approach, and evaluation framework from which to pursue that broader interface. These remain open research problems. If you work on speech recognition, reinforcement learning, diarization, multilingual modeling, or multimodal interfaces, the Wispr Advanced Interfaces Lab is hiring. Check out our latest articles Advancing HCI Wispr is rethinking human-computer interaction — reducing cognitive friction so technology finally feels effortless, intuitive, and built around how you think. Introducing Wispr Advanced Interfaces Lab Wispr CSO Ariya Rastrow introduces the new Wispr Advanced Interfaces Lab. Learn how we're solving the AI interface problem by moving beyond voice-to-voice systems toward intent-driven, context-aware outcomes. Why supporting 100 languages is hard At Wispr Flow, we’re building toward that goal: natural, accurate voice-to-text in 100+ languages. It may sound simple, but it’s one of the hardest technical challenges in AI. Help build the interface between humans and intelligence.