AISimulate FPM Dataset

Forward Pass Model (FPM) libraries and independent latency measurements. This private repository is the input-data source for FPM Gym.

Layout

data/
  <org>--<model>/<system>/<framework>/<framework-version>/<parallelism>/
    manifest.json           # flat metadata for the promoted snapshot
    fpm/                    # optional self-benchmark FPM libraries and sidecars
      provenance/           # FPM producer configurations
    measurements/           # live or independent measurement evidence
      manifest.json
      <truth files>          # manifest role: truth or derived_truth
      provenance/           # configuration, window, and manifest helpers

  history/<org>--<model>/<system>/<framework>/<framework-version>/<parallelism>/
    aisim-commit-<sha-or-unknown-id>/
      manifest.json
      fpm/
      measurements/

catalog/                    # global discovery indexes
schemas/
audit/
scripts/manage_dataset.py

Model directories use the exact case-sensitive Hugging Face model ID with / replaced by --. Example active leaf:

data/MiniMaxAI--MiniMax-M2.7/h200-sxm/vllm/0.25.1/dep4/

Campaign names never define canonical paths. They remain only in provenance. Multi-configuration source Parquets are split by their exact runtime identity, so a parallelism leaf cannot silently contain rows for another configuration. Usable measurement truth is flat under each measurements/ directory. Supporting summaries, configurations, windows, and manifests retain useful source grouping under measurements/provenance/. Only configurations explicitly linked to an FPM artifact are stored under fpm/provenance/.

Each leaf manifest.json is flat. Model, hardware, framework, parallelism, AISim commit, and collector fields apply directly to the promoted data. There are no identity or current wrappers. Its optional history list points to older immutable snapshots.

Measurement-only configurations use fpm: [] and omit the fpm/ directory. They must contain validated measurement truth; an entirely empty or summary-only configuration is not admitted through this path. The catalog records fpm_file_count: 0 independently of measurement availability. Missing predictor data does not invalidate a measured point.

Four AgentX captures are published at standard configuration leaves. See AgentX collection and processing for their exact identities, grouped representation, provenance, and reproducible checks.

Kimi K3 DCP self-benchmark

The TP8+DCP8 self-benchmark configuration adds 669 primary balanced points, 14 synthetic-attention boundary comparator points, and 56 nonuniform prefill points preserved as calibration provenance. The aic_fpm_forward_perf v6 schema records optional DCP separately from independent CP; this configuration explicitly records dcp: 8. Consumers must preserve that identity or report DCP as unsupported. Existing v6 artifacts and the older Kimi TEP8 configuration remain unchanged. Sidecars and catalogs record collection_method and source_kind; these calibration inputs are not independent measurement truth.

Artifact and measurement contracts

FPM Parquets are predictor inputs. The existing AIC libraries use schema name aic_fpm_forward_perf and schema version 6; they do not have a measurement protocol. Their logical schema is schemas/aisim-fpm-forward-v6.schema.json.

The pinned DSV4-Flash, DSV4-Pro, Kimi-K3, and GLM source Parquets contain reviewed dtype annotation errors. During publication, a strict campaign- and model-scoped policy corrects only these declared fields:

• DSV4-Flash: MoE fp8 → w4a8_mxfp4_mxfp8; FMHA fp8 → bfloat16.
• DSV4-Pro: MoE fp8_block → w4a8_mxfp4_mxfp8.
• Kimi-K3: MoE fp8_block → w4a16_mxfp4.
• GLM-5.2-NVFP4: FMHA fp8 → bfloat16.

DSV4-Flash and GLM FMHA resolution is also corrected from checkpoint_native to aic_data_fallback_from_fp8. The importer fails if the campaign, model, or source labels differ from the reviewed policy. It preserves GEMM, weight quantization, latency, workload kind, batch size, token coordinates, row order, and row count. Each affected sidecar records the runtime-dtype-annotation-v1 policy and exact source-to-published labels.

Usable measurement bundles declare one of two semantic protocols:

• forward-pass-measurement-v1:

  • Normalizes exact scheduler observations into one synchronized iteration.
  • Retains one measurement for every contributing rank and caches max_rank_wall_time as the default complete-iteration truth. Consumers may derive another latency policy from the authoritative rank measurements.
  • Distinguishes two source kinds:
    • self_benchmark: a benchmark producer groups ranks by benchmark_id.
    • rank_event_stream: rank-local events are either evaluated directly for one rank, joined using a declared corrected receive-time window, or supplied as pre-grouped rank lists with uploader-provided membership and provenance.
  • producer identifies the component that creates rank-local FPM records. Optional collector provenance identifies a separate capture component; it is required when grouping by listener_receive_time_window.
  • grouping records the membership method and its authority. Consumers do not implicitly join ranks by local counters, file position, or timestamp equality. With pre_grouped, membership is already stored; inferred alignments remain labeled derived and retain their audit provenance.
  • Schema: forward-pass-measurement-v1.schema.json.

• forward-pass-record-v1 contains a reduced, self-contained point: scheduled batch/token coordinates plus observed latency. The current DSV4-Pro derived CSV files use this protocol because their raw streams are external. See schemas/forward-pass-record-v1.schema.json.

Workload kind (prefill, decode, mixed, or empty) is derived per record; it is not part of the protocol ID. Producer names, traffic sources, collection topology, and release labels such as r4, r6pin, and r16 stay in provenance.

Summary-only artifacts cannot recreate predictor input. They set measurement_protocol_id: null and use evidence_format_id, such as wave-summary-v1 or bucket-summary-v1. They remain visible as supporting evidence but are not usable measurements.

See measurement protocols for field meanings, serialization rules, and normalization behavior.

Promotion and history

One reviewed snapshot is promoted for each full configuration. The identity includes model revision, system, framework version, all parallel dimensions, weight quantization, and KV-cache dtype.

• Explicit release lineage decides between generations such as R17/R16 and DSV4-Pro r9/r8/r3 and DEP8 r6pin/r4.
• A later reviewed campaign supersedes an earlier campaign only where their full configuration identities overlap.
• Legitimate variants from one promoted snapshot, such as KV-warm and KV-off, remain together.
• Superseded FPM and measurement artifacts move together under data/history/.
• A valid recorded producer SHA uses aisim-commit-<first-8> as the snapshot directory. Missing producer SHAs use aisim-commit-unknown-<deterministic-content-id> and keep aisim_commit: null; the importer never invents a commit.

The top-level manifest records collected_by and how that attribution was derived. aisim_commit describes the producer code when known. provenance.source_revision separately records the audited AISimPlayground source commit.

FPM Gym evaluation

This repository stores source evidence, not evaluation output. FPM Gym:

• Parses the measurement artifacts for a selected configuration snapshot.
• Runs AISim against the matching FPM artifact at evaluation time.
• Derives measurement and prediction states independently.
• Applies exclusions and computes coverage and accuracy metrics.

No predictions, normalized comparison rows, selector mappings, or accuracy metrics are persisted here. Configuration identity comes from the leaf path and manifest. Workload coordinates come from the FPM and measurement contents.

See the evaluation state policy for the measurement and prediction state tables, exclusion rules, and metric denominators. The trend page can include history by following manifest history references or filtering catalog/index.parquet by snapshot_status.

Publication policy

• Consumers must pin a full Hugging Face commit SHA.
• Discovery starts from catalog/index.json or an active leaf manifest.
• Consumers verify every recorded SHA-256 before loading data.
• FPM Gym records the pinned dataset commit and AISim commit for each evaluation.
• Missing or invalid measurement evidence remains visible as a gap; it is never scored.
• Raw logs, reproduction scripts, reports, redundant copies of rank streams, and wholesale prediction directories are not mirrored.

The repository must remain private. Retained provenance contains internal paths and network metadata, and the source has no approved redistribution license.

Audit notes

• DSV4-Pro raw serving streams are external; normalized score truth and its source hash are retained.
• Usable measurement truth files are flat under measurements/; summaries, serving configs, window maps, and helper manifests are under measurements/provenance/.
• DSV4-Pro benchmark-lane configs are linked from the matching FPM metadata and stored under fpm/provenance/.
• DSV4-Pro TEP8 r9 is active. Diagnostic r8 and superseded r3 evidence are in history.
• DSV4-Pro DEP8 r6pin (worker-pinned prefill lanes, decode rows identical to r4) is active; the r4 library and its evidence are in history.
• MiniMax H200 R17 is active for dep4 and pure-tp4; R16 is history. The later static snapshot is active for H200 tep4.
• MiniMax B200 repaired evidence supersedes the earlier decode-only E9 snapshot.
• MiniMax B200 dep4/c4 is quarantined, and dep4 prefill has no retained measurement evidence.
• GLM and MiniMax H200 static anchor inputs are incomplete.
• GLM-5.2 GB200 DEP16 (data/nvidia--GLM-5.2-NVFP4/gb200/vllm/0.28.0/dep16/) publishes the paired-rows decode and chunk-boundary prefill self-benchmark library with canonical forward-pass-measurement-v1 iterations of the 16/16 disaggregated truth run, one file per engine role, reconstructed at import from the raw rank-event streams with the campaign's per-node clock correction (estimated for prefill; declared zero for decode, where no shared step exceeds 150 ms) and 8 ms half-open window (listener_receive_time_window, exact decimal timestamp arithmetic, every rank record retained, max_rank_wall_time cached). The raw streams, bucketed tables, reports and aiperf exports are supporting evidence under measurements/provenance/. The campaign's published fidelity figures used the median wall over busy ranks; on complete decode steps the maximum exceeds the median by about 1% at the median (1.7% at p90, 8% at p99). Decode is 3.6% and prefill 24.0% step-level MAPE against the 16/16 disaggregated truth (17.2% when all 16 ranks prefill); vLLM 0.28.0 is not a curated aisimulate version slot. Its source dtype labels already carry the runtime identity, so no publish-time correction applies. The superseded 12-row decode pair is discarded.

See source mapping, reconciliation, and validation output.

Validate or refresh

Validation is source-independent:

uv run --with pyarrow --with jsonschema python scripts/manage_dataset.py validate

Rebuild from the audited internal checkout:

uv run --with pyarrow --with jsonschema python scripts/manage_dataset.py import \
  --source-root /path/to/AISimPlayground/playground \
  --replace

The importer rejects any source revision other than the pinned audited commit. It also refuses to overwrite externally contributed measurement-only leaves. Rebuild the base import into a fresh destination, then reapply reviewed external ingestions. This prevents a Playground refresh from silently deleting AgentX data.

For the pinned AgentX package, scripts/ingest_agentx.py performs a collision- checked migration and regenerates the standard catalogs. Grouped-list reading and canonical normalization live in scripts/pre_grouped.py; its validator uses stored membership and does not rerun alignment or impose a receive-time window.

DeepSeek V4.1 GB200/H200/B200 TP4 full and decoder bounded

The PR #304 profile guide covers six independently collected TP4 profiles: GB200, H200 and B200, each full and decoder_bounded. Each has 145 source-qualified native calibration rows and a separate 38-geometry heldout set. The guide lists coverage, heldout accuracy, replay entry points and limitations, including the 12 geometries excluded by the bounded aggregate API and the retained B200 full prefill errors.