ARC

Deterministic hardware-aware LLM runtime and planning stack focused on:

• Production path: Linux + NVIDIA CUDA + NVMe + io_uring/GDS
• Co-design path: macOS (Metal + unified-memory validation loops)

ARC is built to make execution plans explicit, storage-tier decisions observable, and long-context inference reproducible.

---

What ARC currently guarantees

• Real Hugging Face model resolution + download + parse + execution-plan generation.
• Deterministic decode control (--seed, --min-new-tokens, --max-tokens).
• Tiered memory planning (VRAM / RAM / NVMe) with explicit resource descriptors.
• Real sweep harness with pass/fail token-generation checks.

What ARC does not claim yet

• Single-node “hundreds of trillions of parameters” at practical throughput on consumer hardware.
• Universal support for every gated/custom architecture without model-specific bring-up.

Large-scale viability still depends on physical limits: GPU compute, PCIe bandwidth, NVMe throughput/latency, and model architecture sparsity.

---

System requirements

Minimum (development / validation)

• Rust toolchain (cargo, stable toolchain)
• Network access for Hugging Face downloads
• ~30 GB free disk for code + cache + logs (more for real model sweeps)

Linux + NVIDIA (production target)

• Linux kernel with io_uring support
• NVIDIA driver + CUDA runtime available (nvidia-smi, nvcc)
• NVMe SSD with sufficient free space for model shards + KV archives
• Recommended:
  • High RLIMIT_MEMLOCK for pinned-memory paths
  • Filesystem/driver stack compatible with direct I/O and GDS workflows

macOS (co-design / parity path)

• Apple Silicon recommended
• Metal-capable environment
• Sufficient unified memory + NVMe free space

---

Build and test

cargo check
cargo test -p arc-runtime
cargo test -p arc-algorithms

---

CLI command reference (all supported modes)

Binary: arc-core

1) Discovery + validation (default)

cargo run --

Runs hardware discovery, validation, and emits a full report.

2) Discovery-only (no validation)

cargo run -- --discover-only

3) Raw JSON system profile

cargo run -- --json

4) Quiet mode (suppress progress logging)

cargo run -- --quiet

5) Load model from local Hugging Face directory

cargo run -- --load-model /absolute/path/to/model-dir

6) Load model from raw config JSON string

cargo run -- --load-model-json '{"model_type":"llama","num_hidden_layers":32,...}'

7) Full run from Hugging Face model ID

cargo run -- --run HuggingFaceTB/SmolLM2-135M-Instruct

8) Full run with prompt + deterministic sampling controls

cargo run -- --run HuggingFaceTB/SmolLM2-135M-Instruct \
   --prompt "Write one short sentence about deterministic inference." \
   --seed 0 \
   --min-new-tokens 8 \
   --max-tokens 64

9) Full run with ultra-long prompt from file (avoids shell arg limits)

cargo run -- --run HuggingFaceTB/SmolLM2-135M-Instruct \
   --prompt-file /absolute/path/to/prompt.txt \
   --seed 0 \
   --min-new-tokens 8 \
   --max-tokens 64

10) Override runtime context target (bounded by model max)

cargo run -- --run HuggingFaceTB/SmolLM2-135M-Instruct \
   --prompt-file /absolute/path/to/prompt.txt \
   --target-context-tokens 262144 \
   --seed 0 \
   --min-new-tokens 8 \
   --max-tokens 64

11) Tune KV/prefill/session settings for long-context and concurrent runs

cargo run -- --run HuggingFaceTB/SmolLM2-135M-Instruct \
   --prompt-file /absolute/path/to/prompt.txt \
   --target-context-tokens 262144 \
   --max-prefill-chunk 8192 \
   --kv-vram-budget-bytes 4294967296 \
   --kv-ram-budget-bytes 34359738368 \
   --prefix-cache-refs 500000 \
   --session-id 101 \
   --seed 0 \
   --min-new-tokens 8 \
   --max-tokens 64

Flag semantics

• --prompt <text>: prompt string for generation.
• --prompt-file <path>: read prompt text from file (recommended for very long prompts).
• --seed <u64>: deterministic sampling seed.
• --min-new-tokens <n>: enforce minimum generation before EOS can stop decode.
• --max-tokens <n>: cap newly generated token count.
• --target-context-tokens <n>: target context size for KV allocation/scratch sizing, clamped to model max context.
• --max-prefill-chunk <n>: upper-bound prefill scratch sizing (defaults to prompt/context-driven value).
• --kv-vram-budget-bytes <n>: KV VRAM budget override in bytes (otherwise auto-derived from remaining VRAM).
• --kv-ram-budget-bytes <n>: KV RAM budget override in bytes (otherwise auto-derived from detected system RAM).
• --prefix-cache-refs <n>: prefix-cache capacity in KV block references.
• --session-id <u64>: explicit non-zero session identifier for per-session tracking.

---

Progressive sweep harness

Script: scripts/progressive_e2e_sweep.sh

Show plan only

./scripts/progressive_e2e_sweep.sh --dry-run

Run one or more models

./scripts/progressive_e2e_sweep.sh \
   --models HuggingFaceTB/SmolLM2-135M-Instruct,Qwen/Qwen2.5-0.5B-Instruct \
   --prompt "Give one sentence about deterministic sparse inference." \
   --seed 0 \
   --min-new-tokens 8 \
   --max-tokens 64

Run + purge model cache after each success

./scripts/progressive_e2e_sweep.sh \
   --models HuggingFaceTB/SmolLM2-135M-Instruct \
   --seed 0 \
   --min-new-tokens 8 \
   --max-tokens 64 \
   --purge-cache

Sweep options

• --model <id> (repeatable)
• --models <a,b,c> (replaces defaults)
• --prompt <text>
• --min-new-tokens <n>
• --max-tokens <n>
• --seed <u64>
• --log-dir <path>
• --purge-cache
• --dry-run

The harness validates logs for Generated N tokens and fails on missing/zero generation.

---

Strict live-validation mode for Hugging Face config tests

Some integration tests can now run in strict live mode.

• Default behavior: strict live for ungated model checks.
• Offline fallback opt-in: ARC_ALLOW_HF_SKIP=1

Examples:

cargo test -p arc-algorithms --test mega_model_partitioning_tests -- --nocapture

ARC_ALLOW_HF_SKIP=1 cargo test -p arc-algorithms --test mega_model_partitioning_tests -- --nocapture

Live model matrix (no hardcoded architecture numerics)

New strict live matrix test:

cargo test -p arc-algorithms --test live_hf_matrix_tests -- --nocapture

This suite validates:

• resolve_model against live Hugging Face metadata
• real config.json download and parse
• generic model invariants (heads/KV heads/layers/context)
• tier-plan invariants (VRAM + RAM + NVMe fractions sum to 1)

Optional large-model matrix (live metadata/config fetch, best-effort):

ARC_ENABLE_HF_LARGE_MATRIX=1 cargo test -p arc-algorithms --test live_hf_matrix_tests live_hf_large_matrix_opt_in -- --nocapture

Optional custom large model list (comma separated):

ARC_ENABLE_HF_LARGE_MATRIX=1 ARC_HF_LARGE_MATRIX_MODELS="deepseek-ai/DeepSeek-R1,zai-org/GLM-5-FP8" cargo test -p arc-algorithms --test live_hf_matrix_tests live_hf_large_matrix_opt_in -- --nocapture

---

Running large-model bring-up safely

For very large models, use staged bring-up:

1. Confirm discovery/validation is clean.
2. Run one ungated small model end-to-end with deterministic flags.
3. Scale model size gradually while tracking:
   • pinned RAM requirements vs memlock
   • NVMe free space and sustained throughput
   • generated-token continuity (non-zero output)
4. Keep logs in run_logs/ for postmortem and regression comparisons.

---

Linux/NVIDIA large-model execution profile (strict)

Use this profile for real NVMe-streamed runs on Linux + NVIDIA.

Blackwell sm_120 required stack (RTX 50xx)

For cc12.0/sm_120 GPUs (RTX 5060 Ti / 5070 / 5080 / 5090), ARC expects:

• BF16 compute dtype (enforced)
• TRITON_ATTN attention backend (FlashAttention classic is unsupported)
• awq_marlin instead of awq
• no gptq / bitsandbytes (bnb) on sm_120

ARC now applies these rules during --run:

• awq is auto-upgraded to awq_marlin
• float16 dtype is upgraded to bfloat16
• known-broken quantization stacks on sm_120 fail fast with a clear error
• for extreme-scale admission, NVMe capacity is the only hard blocker; PCIe/compute/context limits are surfaced as throughput warnings

For ultra-long context and deep reasoning bring-up, ARC also supports:

• --deep-reasoning-profile (auto floor of 2,000,000 context tokens)
• --allow-capacity-risk (bypass extreme-scale admission blockers; not recommended)

Extreme-scale policy (NVMe-first)

ARC now treats extreme-scale feasibility as an NVMe-footprint problem first:

• Hard block: required model+KV footprint exceeds detected NVMe capacity.
• Warnings only: low PCIe link, low estimated tok/s, context clamping, single-channel RAM, missing io_uring.

This means runs may proceed slowly under constrained bandwidth, but they are not rejected unless NVMe capacity is insufficient.

0) One-time environment checks

nvidia-smi
uname -a
ulimit -l

1) Build + runtime sanity

cargo check
cargo test -p arc-runtime transfer::tests:: -- --nocapture

2) Deterministic smoke run (must pass first)

cargo run -- --run HuggingFaceTB/SmolLM2-135M-Instruct \
   --prompt "Write one concise sentence about deterministic sparse inference." \
   --seed 0 \
   --min-new-tokens 8 \
   --max-tokens 64

3) Progressive scale run (ungated models)

./scripts/progressive_e2e_sweep.sh \
   --models HuggingFaceTB/SmolLM2-135M-Instruct,Qwen/Qwen2.5-0.5B-Instruct \
   --prompt "Give one sentence about deterministic sparse inference." \
   --seed 0 \
   --min-new-tokens 8 \
   --max-tokens 64

4) Long-prompt stress loop (sm_120 focused)

Script: scripts/linux_sm120_stress.sh

./scripts/linux_sm120_stress.sh \
   --model Qwen/Qwen2.5-0.5B-Instruct \
   --iterations 3 \
   --seed 0 \
   --min-new-tokens 16 \
   --max-tokens 128 \
   --prompt-chars 32768

This harness runs repeated --run executions with a generated long prompt, writes per-run logs into run_logs/, and fails fast when:

• generation summary is missing
• generated tokens are zero
• a known-broken sm_120 stack is blocked during runtime checks

5) Million-token / deep-reasoning admission preflight

Use this when preparing trillion-scale or million-token workflows:

cargo run -- --run deepseek-ai/DeepSeek-V3 \
   --prompt-file /path/to/very_long_prompt.txt \
   --min-new-tokens 1000000 \
   --max-tokens 1000000 \
   --deep-reasoning-profile \
   --min-target-tok-s 0.001

If ARC reports a hard blocker (insufficient NVMe capacity for required model+KV footprint), it exits non-zero by default. You can force best-effort mode only for experimentation:

cargo run -- --run deepseek-ai/DeepSeek-V3 \
   --prompt-file /path/to/very_long_prompt.txt \
   --min-new-tokens 1000000 \
   --max-tokens 1000000 \
   --deep-reasoning-profile \
   --allow-capacity-risk

6) Strict run acceptance criteria

• Run completes without panic/error.
• Logs include Generated N tokens with N > 0.
• Decode stats show non-zero TPS and TTFT.
• If overlap is active, logs/report include non-zero double-buffer swap activity.
• No silent degraded I/O behavior in strict transfer tests.
• On sm_120: stack report shows BF16 + TRITON_ATTN and no blocked quantizer.

7) For very large streaming models

• Keep deterministic flags (--seed, --min-new-tokens, --max-tokens) fixed while tuning.
• Increase model size/context in small increments and keep run_logs/ artifacts.
• Track pinned-memory pressure (RLIMIT_MEMLOCK) and NVMe free space before each run.
• Treat arc-execution-plan.json + arc-hardware-resources.json as required diagnostics.

This sequence is the intended path before attempting extreme-scale model bring-up.

---

Operational notes

• If a run is interrupted, logs remain in run_logs/.
• Runtime writes planning artifacts such as:
  • arc-execution-plan.json
  • arc-hardware-resources.json
• On architectures without hardware TSC support in this bridge, timing falls back to monotonic clock instrumentation.

---

Related docs

• docs/progressive-e2e-validation.md
• IMPORTANT_AGENT_CONTEXT.txt