LMCache hybrid Mamba/GDN restore fix

Verified fix for silent KV-cache persistence corruption with hybrid Mamba/GDN + full-attention models in LMCache (0.5.3 through current dev) when running vLLM ≥ 0.26.

Patch: patches/lmcache-053-hybrid-fa.patch · sha256 ec14d5012bef7a86e1ed90c6ada8ee1cbf27a7fe7b921a5ff8f83364efaa7de9 (rev 2)

Deep technical narrative: EXPLAINER.md · Upstream issue: LMCache#4701

2026-08-25: LMCache maintainers opened #4731 against dev, carrying our proposed fix (byte-identical over the fix region) and our regression tests, closing #4701. It is not merged, and no released version contains it — the patch here remains the fix for released versions. See Relationship to upstream.

---

The bug

Symptom. With a hybrid model (Mamba/GDN recurrent layers plus a few full-attention layers), LMCache's persistent (disk-tier) store→restart→restore cycle silently persists only 1 of N kernel pages of the full-attention KV per chunk. The disk tier faithfully stores ~4% of the attention cache with no error; restores after an engine restart report high hit rates (the Mamba group restores fine) but return wrong continuations.

Root cause. Hybrid models force vLLM to pick a unified block size N larger than the attention kernel page size, so each logical KV block of the full-attention group spans N / page_size kernel pages — the group is subpaged. Upstream commit 65c2ae81 ("[Core][MP] Support Mamba/GDN hybrid models (Qwen3.5) (#3613)") added _SubpagedAttentionViewEdit in lmcache/integration/vllm/kv_cache_group_edits.py to re-view exactly this layout at registration time — but the edit only pattern-matches the old 5-dim KV tensor layout. vLLM ≥ 0.26 registers full-attention KV as 4-dim K/V-packed (FlashInfer in our runs), so no edit rule matches, the raw kernel-page geometry flows into kv_layer_groups' compression math, and the FA group is misclassified as slot-compressed ("one kernel page holds the whole block"). Only the first kernel page of each chunk is stored and restored.

On 0.5.3 a second, independent bug compounds the failure: paging a multi-page FA object through the single-page GPU staging buffer mutates the shared MemoryObj.meta.address between in-flight async copies, corrupting the host buffer (~96% zeros, no exception). Current dev has independently hardened this path (size-mismatch ValueError, immutable staging-copy descriptors), so dev fails loudly instead of corrupting — but the feature remains broken until the layout matching is fixed.

Affected versions

kv_cache_group_edits.py is byte-identical across all three:

The intended upstream fix never activates on vLLM ≥ 0.26, so as of 2026-08-21 no LMCache version supports disk-tier persistence for this model class on current vLLM without this patch.

Verified environment

• LMCache 0.5.3 (multiprocess mode, LMCacheMPConnector), vLLM 0.27.1
• Primary configuration: hybrid GDN-class model (verified on Qwen3.8, 27B; GDN + full attention), W4A16 AWQ, TP=2, tokens_per_block=1600, slots_per_block=64 → 25:1 subpaging
• Cross-check configuration: Qwen3.5-0.8B hybrid, FlashInfer, TP=1, N=544 → 17:1 subpaging
• LMCache server: --chunk-size 1600 --separate-object-groups, L1 CPU + L2 local-disk (NVMe)

The bug and the fix are geometry-driven, not configuration-driven: both trigger on tokens_per_block > slots_per_block in an uncompressed (kv_size == 2) group — i.e., on the model class, not on tensor-parallel size, model size, or quantization.

The patch

Four files inside LMCache 0.5.3 — no changes to vLLM (vLLM was correct throughout):

• lmcache/integration/vllm/lmcache_mp_connector.py
• lmcache/v1/kv_layer_groups.py
• lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py
• lmcache/v1/platform/base/cache_context.py

What it changes:

1. Treats a subpaged uncompressed FA group as multi-page: derives the page ratio from tokens_per_block // slots_per_block at runtime in the transfer path (verified at 25:1 and 17:1), sizes the reserved host object for all pages, and expands every logical destination L → L*ratio … L*ratio + ratio-1 on both store and retrieve.
2. Introduces an immutable per-page view object for async staging transfers (own copied metadata, per-page address, narrowed shapes, data_ptr = base + k*page_bytes) so the shared base object's metadata is never mutated while copies are in flight. One helper serves both D2H (store) and H2D (restore).
3. Keeps the tracked block ids in sync with vLLM's authoritative block table (MambaManager with align mode + speculative decoding recycles trailing blocks in place without reporting deltas).

Apply

cd <venv>/lib/python3.12/site-packages
patch -p1 --dry-run < lmcache-053-hybrid-fa.patch   # must be clean
patch -p1 < lmcache-053-hybrid-fa.patch
# restart the LMCache server and the vLLM engine

Base: pristine lmcache==0.5.3 wheel. Verify the download:

shasum -a 256 lmcache-053-hybrid-fa.patch
# ec14d5012bef7a86e1ed90c6ada8ee1cbf27a7fe7b921a5ff8f83364efaa7de9

Revisions

Verification methodology and results

A restore system that is sometimes right is worse than none. Every restore must pass a buried-key exact-answer gate: a deterministic long context containing a unique K-<24 hex> sentence buried mid-context is stored, then the vLLM engine is fully killed and restarted (in the TP=1 run, the LMCache server too), the context is restored from the disk tier only, and the model is asked to reproduce the key. Pass requires all four: exact key character-for-character, external hit ≥ 97%, wall time well below cold prefill, engine healthy afterwards. This is deliberately stricter than same-process cache-reset validation with score-level comparison — which cannot catch this class of failure.

Result: 12/12 PASS. The figures above were measured on rev 2 (sha256 ec14d5012…) on 2026-08-22, applied to a pristine lmcache==0.5.3 install. Per-chunk/per-rank md5 of FA host objects identical between store and restore.

Rev 1 (2026-08-21) passed the same 12/12 suite with the same exact keys and the same hit rates (97.0 / 98.8 / 98.7%). Rev 2 is faster in both arms — including the cold-prefill baseline, which the restore path does not touch — so the difference is machine-level run-to-run variation plus the removal of rev 1's diagnostic instrumentation, not a change in restore behaviour. Rev 1 reference for comparison: restore 2.84/4.11/6.99 s vs cold 6.91/16.29/33.92 s.

Relationship to upstream

Reported upstream 2026-08-22: LMCache#4701 — root cause, repro steps and these results, with the proposed fix attached as a reference patch.

Upstream's intended fix for the layout bug (_SubpagedAttentionViewEdit, commit 65c2ae81, contained in released 0.5.3 and 0.5.4 and unchanged through dev) never activates on vLLM ≥ 0.26 because it only matches the old 5-dim KV layout. Rather than asking upstream to adopt this wheel-level patch verbatim, we proposed fixing it the way 65c2ae81 intended: extend the edit-rule matching in kv_cache_group_edits.py to recognize vLLM ≥ 0.26's 4-dim K/V-packed layouts, so the existing registration-time re-view activates. Our restart/exact-answer verification suite was offered as regression-test methodology. Related upstream issue on the same code path: LMCache#4247.

2026-08-25 — upstream picked it up. LMCache maintainer 982945902 opened LMCache#4731 ("fix(vllm): support packed subpaged attention caches") against dev, closing #4701 and crediting the reproducer, root-cause analysis and reference patch. We did not open that PR.

The PR carries our proposal essentially unchanged. We diffed it: the fix region — from _synthetic_packed_content_size through the _EDITS registry, 18708 chars — is byte-identical to the staged proposal in upstream-proposal/ (sha256 cfbfb0bc8ae8e4de… on both sides), including the _logical_block_ratio shared-validation refactor, the byte-accounting matches() predicate, the NHD/HND token-axis resolution via the kv_layout hint, the hard refusal on an absent hint, and the design-doc paragraph. Our GPU-free regression test file was taken too (214 lines). Upstream's only deltas are typing hygiene in the test file (Any/cast for their mypy config) and unrelated dev drift in an adjacent mamba_cache_mode check.

#4731 is open and unmerged, and it targets dev. No released LMCache version contains the rule, so every claim and measurement above stands unchanged.

That proposal is staged in upstream-proposal/ as a git format-patch against dev @ 1a430626, with a GPU-free unit test. We never submitted it as a PR ourselves; upstream took it from the issue.

It is a different patch from the one above, not a port of it, and it is unverified on hardware. The 0.5.3 patch attacks the transfer path and is GPU-verified 12/12, twice; the proposal extends upstream's edit registry instead — the right shape for upstream, but it has never run on any machine, because we run a 0.5.3 wheel in production and have no dev-based deployment. The hardware-verified artifact in this repo remains the 0.5.3 patch above.

License

Apache-2.0 (see LICENSE) — same license as LMCache, so the patch is trivially adoptable upstream.

---

Maintained by 4rce.com Digital Technologies GmbH · project notes: https://stillpointlab.dev