Meetly Auto Engine, Part 1: Language Detection and Model Routing
How Meetly detects the spoken language on an iPhone and routes each recording to one speech model: Whisper Tiny listens, a pair rule decides, one model writes.
Meetly Auto Engine listens before it transcribes. On the iPhone, Whisper Tiny samples up to 24 parts of a recording, a pair rule decides each part between English and one other language, and the recording goes to one designated model: Parakeet, an IndicConformer pack, the Hinglish model, SenseVoice or Whisper. Recording-level detection rose from 88.0% to 94.0%.

Introduction
Meetly Auto Engine is the part of Meetly that turns a finished recording into a transcript with speakers, a verdict on whether it is worth keeping and, if you allow it, a summary. Transcription and speaker tagging run on the iPhone, and audio is not sent to Meetly or to an AI provider. This three-part series explains how the engine works, with its code and our measurements.
This first part covers the first decision: which language the engine is hearing and which speech model should transcribe it. By deciding between English and one other language instead of 99, recording-level detection rose from 88.0% to 94.0%.
Background: what the engine does between Stop and summary
Meetly is an iPhone, iPad and Mac app that records conversations and transcribes them on the device. Most meeting recorders upload audio to a large multilingual model in a data centre, which is simple and accurate because that model can be as large as you like. Meetly keeps the recording on your devices, which rules out that easy path.
One file in; a transcript, a verdict and a summary out
When you tap Stop, the engine is given one audio file that is already on the device. It returns a transcript where each line has a speaker, a verdict on whether the recording is worth keeping and, for kept recordings, a summary, a title and tags. Figure 1 shows every stage on one page.

Every stage in the green region runs on the iPhone with no network. The yellow region is optional and receives text, never audio. Three ideas run through all of it: every stage is resumable from a checkpoint, only one heavy model is in memory at a time, and when the engine is unsure who spoke or which language was spoken, it leaves the line untagged instead of guessing.
Two inputs become a plan that every stage reads
The engine is told two things about a file. The kind is a conversation, recorded on purpose, or a clip, which a wearable recorder we are testing captured when it heard speech. The model is Auto or Fast. Those two values become a plan, and every later stage reads its behaviour from the plan instead of asking what kind of recording it has.
public enum MeetlyAutoEngine {
/// Clip: captured by always-on. Conversation: recorded on purpose.
public enum Kind: String, Codable, Sendable { case clip, conversation }
/// Fast is Parakeet (no check). Auto listens first and picks the model for the language heard.
public enum Model: String, Codable, CaseIterable, Sendable { case auto, parakeet }
public struct Plan: Equatable, Sendable {
public var kind: Kind
public var model: Model
/// A conversation Meetly joined from clips. Nobody is there to answer a language card.
public var automatic = false
public var checksLanguage: Bool { model == .auto }
/// Clips arrive by the dozen: never stop to ask, never wait for a missing model.
public var asksOnLanguageCard: Bool { kind == .conversation && !automatic }
public var waitsForMissingModel: Bool { kind == .conversation && !automatic }
/// An archive verdict archives a clip; a conversation you chose to record is only tagged Review.
public var archivesOnVerdict: Bool { kind == .clip }
public var savedYouLimit: Float { kind == .clip ? 0.45 : 0.35 }
}
}Each property is a product decision written as code. A conversation is something you chose to record, so the engine may ask you a question and may wait for a model to download. Clips arrive by the dozen while you are doing something else, so they never ask and never wait. An archive verdict removes a clip from view, but a conversation you started is only marked for review. Fast means Parakeet with no language check; Auto is the default for both kinds.
The problem: the phone does not know which language it will hear
Before transcription starts, the engine must pick a speech model, and the right model depends on the language spoken. Asking the person every time is not a product, so the engine has to decide by itself. Three things make that hard on a phone: a small detector confuses neighbouring languages, people mix languages in one conversation, and no model a phone can hold covers 99 languages well.
1. Open-set detection confuses neighbouring languages
The natural detector is the language head of Whisper Tiny, an 81 MB model that returns a probability for each of its 99 languages. We measured it on 15 languages with 200 clips each from FLEURS. Asked to choose among all its languages, it was right 78.2% of the time (Mac host).
The mistakes fall between neighbours. In Table 1, read the last column: a router built on these answers would send one Tamil recording in three to the wrong model.
| Spoken | Correct | What Tiny usually said instead |
|---|---|---|
| Urdu | 20.5% | Hindi |
| Malay | 39% | Indonesian |
| Tamil | 57% | Malayalam |
| Cantonese | 0% | Mandarin; it cannot tell them apart |
2. People mix two languages in one conversation
In practice a recording is in English, in one other language, or a mix of the two, and Hindi and English can share a single sentence. A detector that returns one winner for the whole recording hides that mix, and a router that trusts the winner sends a mixed conversation to a model that only knows half of it.
3. One model for 99 languages is slow and weak on most of them
A phone cannot hold one model that is good at everything. The best small English model cannot transcribe Tamil, and the model that covers 99 languages is slow and weak on most of them. So the engine needs several models, and a reliable way to choose between them before the first word is transcribed.
Our approach: listen first, then route to one model per language
The engine answers the language question in four steps. It samples the recording with Whisper Tiny, restricts every decision to English and one other language, lets the device language break close calls, and combines the parts into one answer with a confidence. That answer selects exactly one speech model, and if that model is not on the phone, a conversation waits for it.
Sample at most 24 parts, so the check never hears more than 12 minutes
Before any transcription, the engine runs Tiny's language head over 30-second parts of the recording, at most 24 of them. A short recording is checked part by part; a three-hour recording is sampled at 24 evenly spaced points, so the check never hears more than 12 minutes.
nonisolated static func detectionParts(totalSeconds: Double) -> [Double] {
guard totalSeconds > 0 else { return [] }
let count = max(1, Int((totalSeconds - 5) / 30) + 1)
guard count > maximumDetectionParts else { return (0..<count).map { Double($0 * 30) } }
let step = (totalSeconds - 30) / Double(maximumDetectionParts - 1)
return (0..<maximumDetectionParts).map { (Double($0) * step).rounded() }
}
nonisolated static let maximumDetectionParts = 24Parts with less than one second of speech are skipped, and for each part the engine keeps the full probability distribution, not only the winner. On an iPhone 14 Pro Max the whole check took 3.6 to 7.3 seconds on recordings of 19 to 87 minutes (iPhone).
Decide between two languages, not ninety-nine
Because a recording is almost always English, one other language or a mix of the two, the engine first finds the recording's other language: the non-English language with the most probability summed over all parts. Then each part is decided between English and that one language only, with its probability renormalised over the two.
/// One part decided between English and other, its probability
/// renormalised over the two. A part that heard neither keeps its own top.
public static func restrict(_ distribution: [String: Double], other: String?) -> Part? {
let folded = folded(distribution)
guard let top = folded.max(by: { $0.value < $1.value || ($0.value == $1.value && $0.key > $1.key) }) else { return nil }
guard let other else { return Part(code: top.key, confidence: top.value) }
let en = folded[english] ?? 0, x = folded[other] ?? 0
guard en + x > 0 else { return Part(code: top.key, confidence: top.value) }
return en > x ? Part(code: english, confidence: en / (en + x)) : Part(code: other, confidence: x / (en + x))
}This reuses the probabilities the detector already produced: no second model and no extra time. Urdu is folded into Hindi first, because Tiny confuses the two.
The device language breaks close calls
A language your device is set to wins the other-language slot if it holds at least a quarter of the leader's mass. It is a prior, not an override: the language still has to be heard in the recording. Added to the pair rule, it lifted recording-level detection from 90.1% to 94.0% (Mac host).
Combine the parts into one answer with a confidence
The parts are then combined into one answer. Confidence is the dominant language's share of the recording multiplied by how sure the detector was on those parts, so a recording that is 90% Hindi at 0.9 average probability scores 0.81. It counts as mixed when the runner-up holds 20% of the parts, and the router only acts alone when the result is not mixed and confidence is at least 0.5.
let total = weights.values.reduce(0, +)
// Ties go to the alphabetically first code, so the result is stable.
let ranked = weights.sorted { $0.value > $1.value || ($0.value == $1.value && $0.key < $1.key) }
guard total > 0, let dominant = ranked.first else { return nil }
let share = dominant.value / total
let detector = confidence[dominant.key].map { $0.sum / $0.weight } ?? 1
let runnerUp = ranked.dropFirst().first.map { $0.value / total } ?? 0
return ConversationLanguage(code: dominant.key, confidence: (share * detector * 1000).rounded() / 1000,
mixed: runnerUp >= mixedShare)Route each language to exactly one speech model
Each language has exactly one designated model, and you download only what you speak. Parakeet covers English and 24 European languages, IndicConformer packs cover 22 Indian languages one per pack, a Hinglish model covers Hindi and English mixed, SenseVoice covers Chinese where enabled, and Whisper takes every other language.
public static func model(for language: String, senseVoice: Bool) -> DesignatedModel {
let code = LanguageModelRecommendation.canonical(language)
if TranscriptionModelCoverage.parakeetLanguageCodes.contains(code) { return .parakeet }
if let pack = IndicSpeechPacks.pack(languageCode: code) { return .pack(pack.id) }
if ChineseSpeechPack.routes(code, enabled: senseVoice) { return .pack(ChineseSpeechPack.modelID) }
return .whisper
}Table 2 lists every model the router can choose, with its download size.
| Engine | Languages | Download |
|---|---|---|
| Parakeet TDT 0.6B | English and 24 European languages | 483 or 632 MB |
| IndicConformer (AI4Bharat) | 22 Indian languages, one pack each | 198 MB each |
| Hinglish (Oriserve) | Hindi and English mixed, Latin script | 161 MB |
| SenseVoice Small | Mandarin and Cantonese, where enabled | 240 MB |
| Whisper Small | Every other language Whisper covers | 491 MB |
| Whisper Tiny | The language check only | 81 MB |
| Speaker models | All languages | 22 MB, inside the app |
Specialist packs beat a general model on accuracy and on speed
On the languages our users speak, a general model loses on both axes at once. Figure 2 compares Whisper Large v3 Turbo, a 631 MB model we only run on the Mac, against the 198 MB packs on the same clips. An error rate above 100% is possible because the measure counts inserted words: the general model wrote more wrong words than were spoken.

The pack is also faster. On a one-hour Hindi podcast it finished in 71 seconds, where Turbo took more than 380 (Mac host); in a 20-clip run its real-time factor was 0.018 against 0.10 to 0.18 for Turbo. This is the rare case where accuracy and time do not trade against each other: the specialist wins both at a third of the size.
For English, Parakeet scored a 5.0% and 6.9% word error rate on two long Earnings-21 calls and 14% to 29% on three AMI meetings recorded around a table (Simulator). Across all 44 Earnings-21 calls, 39 hours of audio, it scored 8.2% (Mac host).
When the model is missing, a conversation waits instead of guessing
Say the engine is confident it heard Gujarati and the Gujarati pack is not installed. The easy thing is to run Parakeet and produce a page of confident nonsense. For a conversation, the engine waits instead, shows which model it needs with a download button, and resumes by itself when the download finishes. A clip never waits, because nobody is watching it.
// Unanswered (the app left, or was not open) or a failed download: the recording
// waits for its model, with the page's Download pill, instead of running Parakeet.
// A clip never waits: an uninstalled language's clip runs Parakeet.
let waiting = route == nil && plan?.waitsForMissingModel != false
? Self.waitingModel(needed: needed, answer: answered) : nil
let ran = route ?? waiting ?? (.parakeet, nil)A second guard works the other way: a job that would send a language with its own pack to Whisper is rerouted to the pack, or waits for it. When the check hears English plus one Indian language, the recording goes to that language's pack, and Hindi with English goes to the Hinglish model. A model you picked yourself is never overruled.
Results
The pair rule turned a detector that was right 78.2% of the time per clip into a router that picks the right model for most recordings, with no second model and no extra time on the phone. Figure 3 shows recording-level detection at each step, measured on FLEURS with the app's own logic on a Mac host.

- Language detection. Restricted to a pair, every language we tested scored 94.5% to 100% except Cantonese, and recordings routed both correctly and confidently went from 66.0% to 79.6% (Mac host).
- Transcription. The 198 MB packs beat Whisper Large v3 Turbo on every Indian language we tested, at a third of the size (Figure 2), and Parakeet scored 8.2% word error rate across 39 hours of earnings calls (Mac host).
What's next in this series
Picking the right model is the first decision. Part 2 follows the recording through chunked transcription that survives interruption, the early keep, review or archive filter, speaker tagging, and the voice memory that names people you tagged. Part 3 covers how the models share memory, the optional cloud summary, where the time goes and what we measured but did not ship.
Conclusion
A phone cannot run one model that is good at every language, so Meetly Auto Engine listens first and then routes. The lesson of this part is that the question you ask a small model matters as much as the model. Asked to choose among 99 languages, Whisper Tiny made neighbour mistakes; asked to choose between two, it became a router we can trust.
Record something in any language you speak. The language step, the model it picked and the speakers it found are shown on the conversation page while it works.
Get Meetly for iPhoneQuestions about Meetly Auto Engine
How does Meetly detect the spoken language on device?
Before transcribing, Meetly runs the language head of Whisper Tiny, an 81 MB model, over up to 24 parts of 30 seconds each, so the check never hears more than 12 minutes of a recording. Asked to choose among all its languages, Tiny was right 78.2% of the time on FLEURS, with mistakes between neighbours such as Urdu and Hindi. So the engine first finds the recording's one non-English language, then decides each part between English and that language only. With this pair rule and the device language, recording-level detection rose from 88.0% to 94.0%. On an iPhone 14 Pro Max the whole check took 3.6 to 7.3 seconds.
Which languages can Meetly transcribe on device?
Meetly listens first and then routes each recording to one model built for its language. Parakeet handles English and 24 European languages. Dedicated IndicConformer packs from AI4Bharat handle 22 Indian languages, one pack per language, and a separate model handles Hindi and English mixed in the same sentence. Chinese can go to SenseVoice where it is enabled, and every other language goes to Whisper. You download only the packs for the languages you speak, so a person who records in English and Marathi carries Parakeet and one Indian pack, not every model the engine knows about.
How accurate is Meetly's on-device transcription?
It depends on the language and the audio. For English, Parakeet scored a 5.0% and 6.9% word error rate on two long earnings calls and 8.2% across all 44 Earnings-21 calls, and 14% to 29% on far-field AMI meetings recorded around a table. For Indian languages, the dedicated packs scored 14.6% on Hindi, 17.8% on Bengali and 26.3% on Tamil on the FLEURS test set, where Whisper Large v3 Turbo scored 34.2%, 86.7% and 71.2%. Each figure in this post states whether it was measured on an iPhone, in the simulator or on a Mac test harness.
What happens if the right language model is not downloaded?
For a conversation you recorded on purpose, Meetly does not fall back to a model that cannot handle the language. If the engine is confident it heard, say, Gujarati and the Gujarati pack is not installed, the recording waits, the conversation page shows which model it needs with a download button, and transcription resumes by itself when the download finishes. Clips from a wearable recorder never wait, because nobody is watching them, so they run on Parakeet. A separate guard reroutes any job that would send a language with its own pack to Whisper, and a model you picked yourself is never overruled.