Meetly
EngineeringNovember 21, 202613 min read

Meetly Auto Engine, Part 2: Speaker Tagging and Voice Memory

How Meetly transcribes long recordings on an iPhone without losing work, filters empty ones, tags who spoke and recognises saved voices without guessing.

Meetly transcribes long recordings on the iPhone in resumable chunks, filters empty ones early, and tags speakers with open pyannote models. It leaves a line untagged rather than guess, and names a saved voice only when exactly one fits. On 16 AMI meetings it labelled 81 of 81 enrolled speakers correctly and accepted none of 1,725 impostors.

Inside Meetly Auto Engine. One audio file in. A tagged transcript out.

Introduction

In Part 1, Meetly Auto Engine listened to a recording on the iPhone, worked out its language with a pair rule and routed it to one speech model built for that language. This part follows the recording through the rest of the on-device work: transcription that survives interruption, an early filter that decides whether it is worth more work, and the speaker tagging and voice memory that put names on lines.

The thread through all four stages: when the engine is not sure, it does less, not more.

Background: from a chosen model to named speakers

After Part 1 the engine knows which model to run. Four stages remain before a transcript is ready to read, and all of them run on the phone with no network: chunked transcription, the early filter, speaker tagging and voice memory. Figure 1 in Part 1 shows where they sit, and each one is resumable from a checkpoint.

The problem: long audio, empty recordings and the wrong name

Each of the four stages has its own way to fail, and every failure is visible to the person reading the transcript. Four challenges shaped the design, and each section below answers one of them, in the order the engine meets them after the speech model has been chosen and loaded.

  1. iOS suspends background apps. A 90-minute recording that restarts from zero after every interruption never finishes.
  2. Not every recording deserves the expensive stages. Speaker work and summaries should not be spent on a television in the next room.
  3. Two models must agree on who said each word. Joining speaker turns to words is where most visible mistakes come from.
  4. A wrong name is worse than no name. If the app says Priya said something that Daniel said, you stop trusting every label.

Transcribing long recordings in resumable chunks

The goal is that an interruption costs seconds, not the whole job. The engine makes an energy pass over the audio in 20 millisecond frames, then plans a cut roughly every five minutes, placed at the quietest frame in the 30 seconds before each mark. Each chunk is then transcribed and saved on its own.

Cut in a pause, so no word is split

MeetlyCore/TranscriptionChunking.swiftswift
let firstFrame = earliest / frameSamples
let lastFrame = min(latest / frameSamples, frameEnergies.count - 1)
var cut = latest
if firstFrame <= lastFrame, firstFrame < frameEnergies.count {
    var quietest = firstFrame
    for frame in firstFrame...lastFrame where frameEnergies[frame] <= frameEnergies[quietest] { quietest = frame }
    // Cut in the middle of the quietest frame.
    cut = min(max(quietest * frameSamples + frameSamples / 2, earliest), latest)
}
chunks.append(.init(index: chunks.count, startSample: start, endSample: cut))
start = cut

Cutting in a pause means no word is split across two chunks, and Parakeet carries its decoder state from one chunk to the next, so the joins do not show in the transcript.

Saved chunks make an interruption cost seconds

Each finished chunk is saved as its own file, named by a digest of its content and verified when read back. After a relaunch the job finds the same audio hash, skips every saved chunk, and redoes only the chunk that was in flight.

Heat is handled like an interruption

Before each chunk the engine reads the thermal state and drops to one worker when the phone is warm, or pauses for 60 seconds when it is critical. In the background it plans 45-second chunks so it can stop cleanly inside the time iOS allows. Figure 1 shows the speed this leaves on a phone.

Bar chart of transcription time on an iPhone 14 Pro Max: 19 minutes of audio in 21.6 seconds, 33 minutes in 49.9 seconds, 87 minutes in 176.6 seconds.
Figure 1: Roughly 30 to 54 times faster than real time on an iPhone 14 Pro Max. In real use the median 11.4-minute conversation is transcribed in 26.3 seconds (Telemetry).

The early filter: keep, review or archive

Right after the words exist, before speaker tagging or a summary, the engine decides whether the recording is worth more work. The answer is keep, review or archive, with a one-line reason the person can read. Rules decide first, an optional model check decides only when the rules have no opinion, and what a verdict does depends on the kind of recording.

Deterministic rules catch empty recordings first

MeetlyCore/MeetlyAutoEngine.swiftswift
public static func ruleVerdict(duration: Double, lines: [String], now: Date = Date()) -> Verdict? {
    let text = lines.map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }.filter { !$0.isEmpty }
    if text.isEmpty { return Verdict(decision: .archive, reason: "No speech found", source: .rule, date: now) }
    if duration.isFinite, duration > 0, duration < minimumSeconds {
        return Verdict(decision: .archive, reason: underTenSeconds, source: .rule, date: now)
    }
    let assessment = TranscriptQualityGate.assess(lines: text.map { .init(text: $0) }, duration: duration)
    guard assessment.hasNoMeaningfulContent else { return nil }
    let reason = switch assessment.reason {
    case .repetitive: "Repeated sounds"
    case .empty: "No speech found"
    default: "No real words"
    }
    return Verdict(decision: .archive, reason: reason, source: .rule, date: now)
}

The quality gate behind that call is deterministic. It strips noise markers and phrases that speech recognisers invent on silence, such as "thanks for watching", and looks for loops: one line making up 60% of the transcript, or fewer than one distinct word in four. Hindi written in Latin letters, mixed scripts and long varied talk are never judged against an English dictionary.

An optional model check that fails open

If the rules have no opinion and you allowed cloud processing, one small model call reads the text and answers in one word with a reason of at most twelve words, capped at 200 tokens. A long transcript is not sent whole: the call gets its start, middle and end. The check fails open, so no consent, no network, a timeout or an unreadable answer all mean keep.

MeetlyCore/Intelligence/MeaningCheck.swiftswift
/// The start, middle and end of the text, within maxCharacters.
public static func sample(_ lines: [String], limit: Int = maxCharacters) -> String {
    let text = lines.map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }.filter { !$0.isEmpty }.joined(separator: "\n")
    guard text.count > limit else { return text }
    let part = limit / 3
    let start = text.prefix(part)
    let middleStart = text.index(text.startIndex, offsetBy: (text.count - part) / 2)
    let middle = text[middleStart...].prefix(part)
    let end = text.suffix(part)
    return [String(start), String(middle), String(end)].joined(separator: "\n…\n")
}

Archive only archives a clip

What a verdict does depends on the kind. Archive removes a clip; a conversation you started is only tagged for review unless it is clearly empty. A decision you make yourself by tapping Keep or Archive is stored with you as its source, and the engine never overrides it. Archived recordings are kept and can be restored.

MeetlyCore/MeetlyAutoEngine.swiftswift
/// What this verdict does to a recording of this kind: archive only archives a clip.
public func effective(for kind: Kind) -> Decision {
    decision == .archive && kind == .conversation && source != .you && !isClearlyEmpty ? .review : decision
}

Speaker tagging: who spoke when

Speaker tagging uses the open pyannote community-1 models, converted to Core ML and run through FluidAudio. They are 22 MB and ship inside the app. A segmentation model finds who speaks in each 10-second window, an embedding model turns each voice into 256 numbers called a voiceprint, and clustering groups voiceprints into people. Windows run one second apart to recover short turns.

A ceiling of 8 speakers blended people together

The library asks for a maximum number of speakers, and ours was 8. Above the maximum it does not report extra people; it re-clusters everyone into exactly that many groups. A recorded earnings call with 19 people came back as 8 "people", each a blend of several. Raising one constant from 8 to 32 fixed it and changed nothing on the small meetings we checked for regressions.

Table 1 shows the two recordings where it mattered. Compare the last two columns: the error rate fell and the share of words under the right person rose.

RecordingPeopleFound, 8 → 32Error rateWords under the right person
Earnings-21 Hershey198 → 1925.8% → 5.4%72.3% → 94.2%
Earnings-21 Monro108 → 935.0% → 6.3%62.7% → 93.5%
Table 1: Diarization error rate counts missed speech, false speech and speaker confusion as a share of speech time. Mac host, the app's own speaker code.

Long recordings are tagged in 30-minute windows

Recordings over 60 minutes are split into windows of about 30 minutes cut at a quiet moment, each checkpointed, then speakers are linked across windows when their voiceprints are within 0.35. On a two-hour meeting the windowed path matched the single pass at a 20.3% error rate using 857 MB instead of 1,016 MB (Mac host). On the iPhone 14 Pro Max, tagging took 41 to 238 seconds for 19 to 87 minutes of audio (iPhone).

Untagged is better than guessed

An untagged line is honest; a line confidently given to the wrong person is a mistake the reader cannot see. So the engine joins words to speakers with explicit thresholds, and when a word or a small group of words does not clearly belong to one person, it leaves them without a speaker rather than pick one.

A word takes a speaker only with clear coverage

A word takes a speaker only if that speaker's turns cover at least 30% of the word and the winner holds at least 60% of the covered time. A single word under 0.3 seconds attributed to someone else is treated as flicker and ignored.

Stray speaker groups fold in only when the match is clear

Clustering often produces a small stray group: forty words that are mostly one real speaker, shown as a mysterious Speaker 5. A group under 60 words and under 1% of the recording is folded into a larger speaker only if its voiceprint is within 0.35 of that speaker and the next closest is at least 0.05 farther away. Otherwise its words are left with no speaker at all.

ActNote/Services/Transcription/DiarizationService.swiftswift
for (id, count) in words where !protected.contains(id) {
    let share = Double(count) / Double(total)
    let isSmall = count < small.words && share < small.share
    let isUnprinted = prints != nil && prints?[id] == nil && count < unprinted.words && share < unprinted.share
    guard isSmall || isUnprinted else { continue }
    result.updateValue(prints?[id].flatMap { own in
        let ranked = words.filter { $0.key != id && $0.value > count }.keys
            .compactMap { other in prints?[other].flatMap { SpeakerVerification.distance($0.centroid, own.centroid) }.map { (other, $0) } }
            .sorted { $0.1 < $1.1 }
        guard let best = ranked.first, best.1 <= fold.distance,
              ranked.count < 2 || ranked[1].1 - best.1 >= fold.margin else { return nil }
        return best.0
    }, forKey: id)
}

Across 37 runs on long meetings this removed all 6 stray speakers, left the share of words under the right person unchanged and kept every real speaker, the smallest of whom had more than 70 words (Mac host). Names you set and lines you assigned by hand are never touched.

Voice memory: naming the people you tagged

Tag a speaker once and Meetly recognises that person in later recordings. It is the feature people notice most and the one where a mistake is least forgivable, so three decisions keep it precise: it learns only from clean speech, it names someone only when exactly one saved voice fits, and we measure the mistake that matters, which is naming the wrong person.

Learn only from clean speech

If two people talk in the same 10-second window, the voiceprint for that window is a blend of both, so the engine throws away any window where anyone else speaks at all. The rest are combined around the medoid, the window most like all the others; windows more than 0.45 from it are dropped, and the print fails if more than half would go.

ActNote/Services/Transcription/DiarizationService.swiftswift
/// Each retained interval is genuine diarizer speech within the embedding window.
/// A window touched by another credible speaker is rejected conservatively.
nonisolated static func evidence(from result: DiarizationResult) -> [SpeakerEvidence] {
    (result.chunkEmbeddings ?? []).flatMap { chunk -> [SpeakerEvidence] in
        let overlapping = result.segments.filter {
            min(chunk.endTimeSeconds, Double($0.endTimeSeconds)) > max(chunk.startTimeSeconds, Double($0.startTimeSeconds))
        }
        guard !overlapping.contains(where: { $0.speakerId != chunk.speakerId }) else { return [] }
        return overlapping.filter { $0.speakerId == chunk.speakerId }.map { turn in
            SpeakerEvidence(speakerID: chunk.speakerId,
                start: max(chunk.startTimeSeconds, Double(turn.startTimeSeconds)),
                end: min(chunk.endTimeSeconds, Double(turn.endTimeSeconds)), embedding: chunk.embedding256,
                observationStart: chunk.startTimeSeconds, observationID: String(chunk.chunkIndex))
        }
    }
}

A print needs 3 seconds of clean speech and a saved voice needs 15 seconds before it names anyone. The cost is real: someone who only speaks in quick back-and-forth may never get a clean window. On far-field meeting audio, 60 of 63 people got a print.

Name someone only when exactly one saved voice fits

The matching code reads best as a list of reasons not to name anyone.

MeetlyCore/SavedVoices.swiftswift
var result: [Int: SavedVoice] = [:]
for (speaker, ranked) in distances where !excluding.contains(speaker) {
    guard let best = ranked.first, best.1 <= limit(best.0),
          ranked.filter({ $0.1 <= limit($0.0) }).count == 1 else { continue }
    if ranked.count > 1, ranked[1].1 - best.1 < policy.margin { continue }
    let rivals = distances.filter { other, list in
        other != speaker && list.contains { $0.0.id == best.0.id && $0.1 <= limit($0.0) }
    }
    // On a device you wear, several speakers near your own voice are you, split by the
    // diarizer; they all match you. Other saved people still abstain on a rival.
    guard rivals.isEmpty || (best.0.isSelf && selfThreshold != nil) else { continue }
    result[speaker] = best.0
}
return result

Read it as four refusals. The nearest saved voice must be within 0.35 in cosine distance. Exactly one saved voice may be inside that limit. The runner-up must be at least 0.05 farther away. And if two speakers in one recording both match the same saved voice, neither is named, because one of them must be wrong. Saved voices only name speakers after clustering; they never change who is found.

Measure the mistake that matters: a wrong name

We taught voices from one AMI meeting per group and tested on the other three. In Table 2 the column to watch is the last one: how often a person was given someone else's name.

AudioNamed rightAbstainedNamed wrong
Clean headset mix41 of 4100
Call quality (12 kbit/s Opus)40 of 4110
Everyone enrolled, some absent81 of 8100
1,725 impostor trialsn/an/a0 accepted
Table 2: AMI corpus, 16 meetings, 63 people; voices taught from one meeting per group and tested on the other three. Mac host.

The one miss on call audio was an abstention, not a wrong name. The closest any stranger came to a saved voice was 0.396 on phone audio, so the 0.35 limit has room. With a library of 30 saved voices over 10,000 trials, 99.7% of labels were correct and 0.17% of strangers were labelled (Mac host).

Your own voice on a wearable gets a wider limit

One limit differs. On clips from a recorder you wear, the limit for your own voice is 0.45. On short noisy clips our own voice measured 0.13 to 0.49 from its saved print and nobody else came nearer than 0.6, and the device is on your body. Every other saved voice keeps 0.35. A voiceprint is 256 numbers with no audio kept; saved voices sync only through your private iCloud, encrypted, if you turn sync on.

Results

In real use the median 11.4-minute conversation is transcribed in 26.3 seconds (Telemetry). Raising the speaker ceiling from 8 to 32 cut the error rate on a 19-person call from 25.8% to 5.4%, the stray-speaker rule removed all 6 strays across 37 runs, and voice memory named 81 of 81 enrolled people correctly while accepting none of 1,725 impostors (Mac host).

What's next in this series

So far everything has run on the iPhone. Part 3 covers how it all fits on a phone and what leaves it: the scheduler that keeps one heavy model in memory at a time, what the optional cloud summary sends and refuses, where the time goes from Stop to summary, and what we measured but did not ship.

Conclusion

The four stages in this part solve different problems, but they share one rule. When the engine cannot be sure, it chooses the outcome a person can recover from: redo one chunk, keep a recording, leave a line untagged or leave a voice unnamed. That rule costs some coverage, and it is why the labels that do appear can be trusted.

Record something in any language you speak. The language step, the model it picked and the speakers it found are shown on the conversation page while it works.

Get Meetly for iPhone

Questions about Meetly Auto Engine

How does Meetly know who is speaking?

Open pyannote speaker models, converted to Core ML and bundled in the app, find who speaks when and turn each voice into a voiceprint of 256 numbers. When you tag a speaker, Meetly learns that voice from clean speech only, meaning windows where nobody else talks. It names a speaker later only when exactly one saved voice is within the distance limit, the next closest voice is clearly farther away, and no other speaker in the same recording also matches. On our test set this gave 81 of 81 correct labels and accepted none of 1,725 impostors; when it is unsure it leaves the speaker unnamed.

What happens if iOS interrupts a long transcription?

An interruption costs seconds, not the whole job. Meetly makes an energy pass over the audio in 20 millisecond frames and cuts it roughly every five minutes at the quietest frame in the 30 seconds before each mark, so no word is split. Each finished chunk is saved as its own file, named by a digest of its content and verified when read back. After a relaunch the job finds the same audio hash, skips every saved chunk and redoes only the one that was in flight. The engine also drops to one worker when the phone is warm and plans 45-second chunks in the background.

Does Meetly delete recordings it thinks are empty?

No. Right after transcription the engine gives each recording a verdict of keep, review or archive with a one-line reason. Archive removes a wearable clip from view, but a conversation you started yourself is only tagged for review unless it is clearly empty. Archived recordings are kept and can be restored. If you tap Keep or Archive, that decision is stored with you as its source and the engine never overrides it. The optional model check fails open: no consent, no network, a timeout or an unreadable answer all mean keep.

Why does Meetly sometimes leave a line without a speaker?

Because an untagged line is honest and a line given to the wrong person is a mistake the reader cannot see. A word takes a speaker only if that speaker's turns cover at least 30% of the word and the winner holds at least 60% of the covered time. A small stray group under 60 words and under 1% of the recording is folded into a larger speaker only when its voiceprint is clearly closest to that speaker; otherwise its words stay untagged. Across 37 runs on long meetings this removed all 6 stray speakers and kept every real one.