LLMLingua-Swift

On-device LLM prompt compression for macOS and iOS. A Swift package that wraps a CoreML-converted BERT classifier to compress prompts before sending them to a local or context-limited LLM.

Based on LLMLingua-2 (Microsoft Research). No training required — the pre-trained model is downloaded from HuggingFace and converted once.

What it does

Given a prompt that's too long for your model's context window, LLMLingua-Swift removes low-salience words while preserving meaning. The output is not grammatical prose — it's a compressed token sequence that an LLM can still understand.

Input:  "The meeting was called to order at 9 AM. The agenda included a review
         of last quarter results, discussion of the upcoming product launch, and
         allocation of budget for the next fiscal year."

Output: "meeting 9 AM agenda included review last quarter results upcoming
         product launch allocation budget next fiscal year"

Ratio:  2.2x compression at rate=0.5

Requirements

• macOS 13+ or iOS 16+
• Xcode 15+

Setup

The CoreML model (~171 MB) is not bundled in the package. Download it once and embed it in your app bundle — no runtime network calls needed.

1. Add the package

In Xcode choose File › Add Package Dependencies and enter the repository URL, or add it to Package.swift:

.package(url: "https://github.com/nburns/llmlingua-swift", from: "0.1.0")

2. Download the model

Run the SPM command plugin from your project directory:

swift package plugin --allow-writing-to-package-directory download-model

This downloads llmlingua2.mlpackage (~171 MB) and vocab.txt into a Model/ directory. It's a one-time step — subsequent builds skip the download if the files are already present.

Alternatively, use the shell script directly:

REPO_OWNER=nburns ./scripts/download_model.sh

3. Add the model to your Xcode target

Drag Model/llmlingua2.mlpackage into your Xcode project. In the Add Files dialog, check Copy items if needed and select your app target.

That's the only file you need to add. The vocabulary (vocab.txt) is bundled inside the LLMLingua Swift package and loaded automatically.

Xcode compiles the .mlpackage to .mlmodelc at build time. The model is embedded in the app bundle — no downloads at runtime.

4. Compress prompts

import LLMLingua

// Loads llmlingua2.mlmodelc and vocab.txt from Bundle.main
let compressor = try Compressor(bundle: .main)

let result = try await compressor.compress(
    context: ["Your long prompt here..."],
    rate: 0.5   // keep ~50% of tokens
)

print(result.compressedPrompt)
print("Compressed \(result.originTokens) → \(result.compressedTokens) tokens (\(String(format: "%.1fx", result.ratio)))")

Compression parameters

Multiple segments

let result = try await compressor.compress(
    context: [
        "Background document...",
        "Retrieved passage...",
    ],
    rate: 0.5
)
// result.compressedPromptList — one string per segment
// result.compressedPrompt    — segments joined with "\n\n"

Running tests

Tokenizer unit tests need no model:

swift test --filter TokenizerTests

Integration tests validate against Python-generated fixtures (requires the model):

LLMLINGUA_MODEL_PATH=/path/to/llmlingua2.mlpackage \
LLMLINGUA_VOCAB_PATH=/path/to/vocab.txt \
swift test --filter CompressorTests

How it works

See the documentation or AGENTS.md for a description of the compression pipeline and implementation decisions.

Known limitations

• Ratio counts differ: Swift uses BERT token counts; the Python reference uses tiktoken. Same kept words, different ratios. Test tolerance is ±15%.
• Tiktoken approximation: Threshold weighting uses max(1, word.count / 4) — tiktoken isn't practical to bundle for on-device use.
• No multi-word force tokens: forceTokens matches single words only.

See TODO.md for more.

Publishing a new model release

If you need to re-convert the model (e.g. after updating coremltools), run the release workflow:

git tag models-v2
git push origin models-v2

This triggers .github/workflows/release-model.yml, which converts the model on a macOS runner and uploads the artifact to the GitHub Release. Update defaultTag in Plugins/DownloadModel/plugin.swift to point consumers at the new release.