AxiiDiarization

Native Swift speaker diarization using CoreML and Accelerate. Runs on-device on macOS 14+ and iOS 17+ with zero external dependencies.

• 5.3% DER on VoxConverse (10s windows)
• ~400x real-time on Apple Silicon
• Batch and streaming modes
• Speaker identity tracking (pin, auto-match, enriched embeddings)
• Speaker matching across recordings

Installation

Add to your Package.swift:

dependencies: [
    .package(url: "https://github.com/AXI-Labs/AxiiDiarization.git", from: "1.0.0"),
]

Prerequisites

• macOS 14+ / iOS 17+
• Xcode (for building)
• Python 3 (for model conversion and evaluation)

Model Conversion (one-time)

Two CoreML models must be converted from their source checkpoints:

# Install NeMo environment
python3 -m venv .venv_nemo
.venv_nemo/bin/pip install nemo_toolkit[asr] coremltools

# Sortformer v2.1 segmentation (batch=4, 10s windows)
.venv_nemo/bin/python3 eval/convert_sortformer_coreml.py --batch-size 4 --mel-frames 1024

# ResNet34 speaker embeddings
.venv_nemo/bin/python3 eval/convert_resnet34_coreml.py

This creates:

• Models/sortformer_4spk_v21.mlpackage (441 MB, NVIDIA Open Model License)
• Models/wespeaker_resnet34.mlpackage (25 MB, Apache 2.0)

Conversion options: --batch-size (default 4), --mel-frames (1024 for 10s windows, 3000 for 30s).

Build

# Library + CLI (from Tools/)
cd Tools && xcodebuild -scheme DiarCLI -derivedDataPath ../.build/xcode -destination 'platform=macOS' -configuration Release build

# Binary location
.build/xcode/Build/Products/Release/DiarCLI

Use Release builds for evaluation and benchmarks. Debug builds are 5-10x slower.

Python Setup (for evaluation)

python3 -m venv .venv
.venv/bin/pip install -r eval/requirements.txt

Usage

CLI=.build/xcode/Build/Products/Release/DiarCLI

# Full diarization pipeline (default models, 10s window recommended)
$CLI run <audio.wav> --window-duration 10

# Custom model paths
$CLI run <audio.wav> --sortformer-model Models/sortformer_4spk_v21.mlpackage --emb-model Models/wespeaker_resnet34.mlpackage

Audio must be 16kHz mono 16-bit PCM WAV.

Speaker Profiles

# Run with profile matching
$CLI run podcast.wav --profiles profiles.json

# Name a speaker from the last run
$CLI profile set --profiles profiles.json --speaker SPEAKER_00 --name "Alice"

# List known profiles
$CLI profile list --profiles profiles.json

Re-run on a different recording with the same --profiles file to auto-match speakers.

Library API

Batch Diarization

let pipeline = try DiarizationPipeline(
    sortformerModelPath: "Models/sortformer_4spk_v21.mlpackage",
    embModelPath: "Models/wespeaker_resnet34.mlpackage",
    windowDuration: 10.0
)
let result = pipeline.run(audio: samples)  // [Float] at 16kHz
for seg in result.segments {
    print("\(seg.speaker.label): \(seg.start)s - \(seg.end)s")
}

Streaming (Incremental)

let session = pipeline.createSession()
session.addAudio(chunk1)
let partial = session.process()   // segments so far
session.addAudio(chunk2)
let updated = session.process()   // re-clustered with all data
let final = session.finalize()

Speaker Identity Tracking

Streaming sessions track speaker identity across process() calls. The library handles re-clustering internally — apps work with stable speaker identities, not raw cluster labels.

// Define your profile type (library only needs id + embeddings)
struct MyProfile: SpeakerProfile, Codable {
    var id: String
    var name: String            // app-owned metadata, not part of protocol
    var embeddings: [[Float]]
}

// Pre-load known speakers into a session
let saved: [MyProfile] = loadFromDisk()
let session = pipeline.createSession(knownSpeakers: saved)

session.addAudio(chunk)
let result = session.process()

// Each segment tells you how the speaker was identified
for seg in result.segments {
    switch seg.identification {
    case .unknown:
        print("\(seg.speaker.label): \(seg.start)s - \(seg.end)s")
    case .autoMatched(let speakerID, let confidence):
        let name = saved.first { $0.id == speakerID }?.name ?? speakerID
        print("\(name) (\(Int(confidence * 100))%): \(seg.start)s - \(seg.end)s")
    case .pinned(let speakerID):
        let name = saved.first { $0.id == speakerID }?.name ?? speakerID
        print("\(name) [pinned]: \(seg.start)s - \(seg.end)s")
    }
}

// Pin segments to a speaker (captures embeddings, survives re-clustering)
let snapshot = result.segments
    .filter { $0.speaker.label == "SPEAKER_02" }
    .map { ($0.start, $0.end) }
session.pinSegments(snapshot, toSpeaker: myProfile)

// Export enriched embeddings after session
let final = session.finalize()
if let embs = final.enrichedEmbeddings(for: myProfile) {
    myProfile.embeddings = embs
    saveToDisk(myProfile)
}

Speaker Matching (Standalone)

For batch (non-streaming) diarization, match speakers against profiles directly:

let matches = SpeakerMatcher.match(speakers: result.speakers, against: profiles)
for match in matches {
    if let profile = match.profile {
        print("\(match.speaker.label) → \(profile.name) (confidence: \(match.confidence))")
    }
}

// Embedding utilities
let sim = SpeakerMatcher.cosineSimilarity(a, b)
let centroid = SpeakerMatcher.centroid(from: embeddings)

Architecture

Sources/AxiiDiarization/
  Audio/FBank.swift              Kaldi-compatible mel filterbank (80 bins, Accelerate/vDSP)
  Audio/NeMoFBank.swift          NeMo-compatible mel filterbank for Sortformer (128 bins)
  Models/ResNet34CoreML.swift    ResNet34 CoreML wrapper (Apple Neural Engine)
  Models/SortformerSegmentation.swift  Sortformer v2.1 CoreML segmentation
  Pipeline/DiarizationPipeline.swift   Types, clustering, reconstruction
  Pipeline/DiarizationPipeline+Steps.swift  Segmentation + embedding extraction
  Pipeline/DiarizationSession.swift    Streaming session (incremental processing)
  Pipeline/Clustering.swift      AHC with centroid linkage
  Speaker/SpeakerMatcher.swift   SpeakerProfile protocol + matching + embedding math

Tools/Sources/DiarCLI/
  DiarCLI.swift                  CLI (run, profile)

Tools/Sources/DiarUI/
  DiarUIApp.swift                SwiftUI app with AppDelegate window activation
  AudioService.swift             Audio loading (any format) + AVAudioPlayer playback
  TimelineView.swift             Canvas-based segment visualization with playhead
  DiarUIViewModel.swift          Pipeline + streaming + FluidAudio transcription + sync
  ContentView.swift              Toolbar, timeline, transcript, speaker panel

Tests/AxiiDiarizationTests/
  ClusteringTests.swift          AHC clustering unit tests
  HelperTests.swift              Pipeline helper tests (activationsToSegments, maskAudio)
  SpeakerMatcherTests.swift      Matching + embedding math tests
  SpeakerTrackingTests.swift     Identity tracking (pin, auto-match, enriched embeddings)

eval/
  eval_swift.py                  DER benchmark against VoxConverse
  eval_embeddings.py             Within-recording embedding quality (EER)
  eval_cross_recording.py        Cross-recording speaker ID with AMI corpus
  convert_resnet34_coreml.py     ResNet34 -> CoreML conversion
  convert_sortformer_coreml.py   Sortformer -> CoreML conversion
  requirements.txt               Python dependencies

Evaluation

Place 7 VoxConverse test files (.wav + .rttm) into TestData/{simple,medium,hard}/. These are gitignored due to size.

DER Benchmark

# All 7 files with 10s window (recommended)
.venv/bin/python3 eval/eval_swift.py --window-duration 10

# Quick smoke test, specific files, save baseline
.venv/bin/python3 eval/eval_swift.py --category simple
.venv/bin/python3 eval/eval_swift.py --files akthc,ampme
.venv/bin/python3 eval/eval_swift.py --save-baseline

Results (Sortformer v2.1, 10s window, CoreML):

Window duration comparison:

Embedding Quality

# Within-recording speaker verification (EER)
python3 eval/eval_embeddings.py

# Cross-recording speaker identification (AMI corpus)
python3 eval/eval_cross_recording.py --max-groups 10

The cross-recording evaluation uses the AMI Meeting Corpus Mix-Headset audio (all microphones mixed), representing a worst case for speaker separation. The script loads actual participant IDs from meetings.xml to handle label inconsistencies across meetings.

Tests

xcodebuild test -scheme AxiiDiarization -derivedDataPath .build/xcode -destination 'platform=macOS'

50 tests covering clustering, pipeline helpers, speaker matching (protocol, greedy assignment, embedding math), and speaker identity tracking (identification types, enriched embeddings, registered speakers). No model files required.

DiarUI Validation App

SwiftUI app for validating diarization quality. Loads any audio format (MP3/M4A/WAV), visualizes speaker segments on a timeline, plays audio with synchronized transcript scrolling, and transcribes using FluidAudio (Parakeet TDT v3).

# Build (use Release for performance)
cd Tools && xcodebuild -scheme DiarUI -derivedDataPath ../.build/xcode -destination 'platform=macOS' -configuration Release build
.build/xcode/Build/Products/Release/DiarUI

Features:

• Load MP3/M4A/WAV/AIFF via AVFoundation
• Streaming diarization with progressive timeline updates
• Per-segment transcription with Parakeet TDT v3
• Synchronized playhead: timeline, transcript, and speaker panel all track current position
• Click transcript line or timeline to seek; drag slider to scrub
• Speaker identity tracking: visual distinction between unknown, auto-matched, and pinned segments
• Rename speakers to pin identity (with confidence display for auto-matched)
• Unpin speakers to revert to auto-detection
• Right-click transcript lines to reassign segments to a different speaker
• Profile persistence with enriched embeddings for cross-file matching

First run downloads the Parakeet model (~600MB). Profiles are saved to ~/Library/Application Support/DiarUI/profiles.json.

License

MIT License. See LICENSE for details.

Model weights have separate licenses:

• Sortformer v2.1: NVIDIA Open Model License
• WeSpeaker ResNet34: Apache 2.0