Memoir JSA Submission Artifact

Repository: https://github.com/Rhys-Wang-wannaLearnMath/Memoir_JSA_Submission_Artifact

This repository is the reviewer-facing artifact package for Memoir, a WebAssembly-based LLM inference system with compile-time selectable KV-cache policies. It is organized for source inspection, data verification, figure traceability, and long-term archival.

Large external model weights are not included. When rerunning examples, place model files locally and configure model directories as described in 01_Memoir_Code_and_Artifacts/README.md.

Top-Level Layout

Memoir_JSA_Submission_Artifact/
├── 00_REVIEWER_INDEX.md
├── README.md
├── ENVIRONMENT.md
├── MODEL_WEIGHTS.md
├── LICENSE
├── CITATION.cff
├── CHECKSUMS.txt
├── requirements.txt
├── 01_Memoir_Code_and_Artifacts/
├── 02_RQ1_RQ3_Consolidated_Data/
├── 03_RQ2_Sensitivity_Analysis_Data/
├── 04_Memory_Layout_Analysis_Data/
├── 05_Paper_Figures/
└── 06_RQ4_Baseline_Comparison_Data/

For a fast reviewer walkthrough, start with 00_REVIEWER_INDEX.md.

Directory Overview

Research-Question Mapping

The artifact is organized around the paper's experimental questions.

01: Code and Artifacts

01_Memoir_Code_and_Artifacts/ contains the implementation and executable WASM artifact bundle.

01_Memoir_Code_and_Artifacts/
├── README.md
├── sources/
│   ├── README.md
│   └── candle_memoir/
├── wasm_used/
│   ├── qwen3_baseline.wasm
│   ├── qwen3_memoir.wasm
│   ├── smol_baseline.wasm
│   ├── smol_memoir.wasm
│   ├── qwen_baseline.wasm
│   └── qwen_memoir.wasm
└── runtime_source_used/
    ├── README.md
    ├── hooks/
    └── sources/

The source tree uses one implementation with compile-time policy selection:

Relevant implementation files include:

• KV subsystem: 01_Memoir_Code_and_Artifacts/sources/candle_memoir/candle-transformers/src/kv_memory/
• Qwen1.5 integration: 01_Memoir_Code_and_Artifacts/sources/candle_memoir/candle-transformers/src/models/qwen2.rs
• Quantized Qwen3 integration: 01_Memoir_Code_and_Artifacts/sources/candle_memoir/candle-transformers/src/models/quantized_qwen3.rs
• Quantized SmolLM2 integration: 01_Memoir_Code_and_Artifacts/sources/candle_memoir/candle-transformers/src/models/quantized_llama.rs

Model Assets

Model weights are external. The focused examples use environment variables instead of machine-specific paths:

See 01_Memoir_Code_and_Artifacts/README.md for a minimal Wasmtime run template.

02: RQ1 and RQ3 Consolidated Data

02_RQ1_RQ3_Consolidated_Data/ aggregates cross-model results and summary scripts for RQ1 and RQ3.

02_RQ1_RQ3_Consolidated_Data/
├── README.md
├── model_qwen1_5_rq1_rq3/
├── model_quantized_smollm2_rq1_rq3/
├── model_quantized_qwen3_rq1_rq3/
├── combine_interval_improvement.py
├── combine_scheduling_comparison.py
├── combined_speed_comparison.py
└── combined_output/

Each model directory includes runtime-grouped results and analysis assets. The common runtime grouping is:

NATIVE/
WAMR/
WASMTIME/
WAVM/

These directories contain raw generation outputs, token-level performance files, repeat-analysis files, model-info files, and quality-evaluation outputs where applicable. The root-level aggregation scripts combine model-level data into cross-model and cross-runtime summaries.

Important combined outputs include:

• combined_output/combined_all_runtimes_interval_improvement.pdf
• combined_output/combined_all_runtimes_speed_comparison.pdf
• combined_output/combined_all_runtimes_scheduling_comparison.pdf

03: RQ2 Sensitivity Analysis Data

03_RQ2_Sensitivity_Analysis_Data/ contains the RQ2 sensitivity analysis for Memoir parameters, including cache capacity, ratio, throughput, repetition rate, and output quality.

03_RQ2_Sensitivity_Analysis_Data/
├── README.md
├── runtime_wamr/
├── runtime_wasmtime/
├── runtime_wavm/
├── quality_eval/
├── output_rq2_figures/
├── output_rq2_quality/
├── analyse_rq2_quality_scores.py
├── plot_rq2_repeat_patterns_horizontal.py
└── plot_rq2_repetition_rate_vs_capacity.py

Runtime data

The runtime directories contain raw experiment outputs for the WAMR, Wasmtime, and WAVM runtimes across Memoir parameter configurations. Configuration directory names encode the ratio and total cache capacity, for example:

qwen_Memoir_penalty..._RATIO2toR..._totalCapacity.../

Typical files include:

• heuristic_text_generation_results.jsonl: raw generated output and repeat flags
• heuristic_text_generation_speed_info.json: extracted speed information
• three_runs/: three independent runs for the configuration

Quality evaluation data

quality_eval/ contains LLM-judge quality scores produced through the HelloBench/HelloEval-style evaluation pipeline. It covers 70 Memoir configurations plus one Original baseline configuration. Evaluation scores use a 0-10 range and are used by the quality-analysis scripts.

Some raw quality-evaluation JSONL files contain natural-language judge comments. These comments evaluate the quality of model-generated answers and may mention issues such as unsupported or fabricated claims in the model output. Such comments describe generated-answer quality, not the provenance of this artifact's experimental data.

RQ2 output directories

• output_rq2_figures/: repetition-rate figures and statistics
• output_rq2_quality/: quality-score tables, reports, and plots

The main scripts are:

python plot_rq2_repeat_patterns_horizontal.py
python plot_rq2_repetition_rate_vs_capacity.py
python analyse_rq2_quality_scores.py

Run the scripts from inside 03_RQ2_Sensitivity_Analysis_Data/ so relative paths resolve as expected.

04: Memory Layout Analysis Data

04_Memory_Layout_Analysis_Data/ contains statistical analysis assets for memory-layout gaps across models under NATIVE and Wasmtime.

04_Memory_Layout_Analysis_Data/
├── README.md
├── model_qwen1_5/
├── model_quantized_smollm2/
├── model_quantized_qwen3/
├── analyze_memory_layout_all_models.py
└── analysis_results/

The unified analysis script is:

python3 analyze_memory_layout_all_models.py

The analysis_results/ directory contains generated CSV and Markdown report outputs.

05: Paper Figures

05_Paper_Figures/ is a figure-only directory aligned with the submitted paper.

05_Paper_Figures/
├── README.md
├── Method/
│   ├── Method1.pdf
│   ├── Method2.pdf
│   ├── Method3.pdf
│   └── Method4.pdf
└── Experiments/
    ├── RQ1/
    ├── RQ2/
    ├── RQ3/
    ├── RQ4/
    └── case_study.pdf

This directory is intended as the clean figure collection for paper-to-artifact traceability. It contains only the current paper figures and excludes historical or archive-style figure outputs.

06: RQ4 Baseline Comparison Data

06_RQ4_Baseline_Comparison_Data/ contains RQ4 experiment-ready data and analysis outputs.

06_RQ4_Baseline_Comparison_Data/
├── README.md
├── rq4_tasks.jsonl
├── combine_baseline_comparison.py
├── generate_rq4_evidence_metrics.py
├── quality_eval/
│   ├── llm_judge.py
│   ├── summary_scores.json
│   └── eval_results/
├── analysis/
│   ├── rq4_evidence_report.md
│   ├── rq4_metrics.json
│   └── figures/
└── <Model>/<RUNTIME>/<Algorithm>/

Key generated artifacts include:

• analysis/figures/combined_all_runtimes_baseline_comparison.pdf
• analysis/figures/qwen_all_runtimes_baseline_comparison.pdf
• analysis/figures/smollm2_all_runtimes_baseline_comparison.pdf
• analysis/figures/qwen3_all_runtimes_baseline_comparison.pdf
• analysis/figures/combined_all_runtimes_baseline_summary.md
• analysis/figures/hellobench_quality_summary.md

Regeneration entry points:

cd 06_RQ4_Baseline_Comparison_Data
python3 quality_eval/llm_judge.py
python3 generate_rq4_evidence_metrics.py
python3 combine_baseline_comparison.py

Recommended Review Workflow

A reviewer can inspect the artifact in the following order:

1. Read this top-level README.md for the package map.
2. Inspect 01_Memoir_Code_and_Artifacts/README.md for implementation layout and model-file requirements.
3. Inspect the unified source tree under 01_Memoir_Code_and_Artifacts/sources/candle_memoir/.
4. Use 02_RQ1_RQ3_Consolidated_Data/ to verify RQ1/RQ3 data, scripts, and combined figures.
5. Use 03_RQ2_Sensitivity_Analysis_Data/ to verify RQ2 sensitivity data, quality analysis, and repetition-rate analysis.
6. Use 04_Memory_Layout_Analysis_Data/ to verify memory-layout statistics.
7. Use 06_RQ4_Baseline_Comparison_Data/ to verify RQ4 data, quality scores, and generated comparison figures.
8. Use 05_Paper_Figures/ to locate the figures corresponding to the paper.

Reproducing Analysis Outputs

The artifact focuses on data and script traceability. Most plotting and aggregation scripts are designed to be run from their containing directories.

Example workflow:

cd 02_RQ1_RQ3_Consolidated_Data
python combined_speed_comparison.py
python combine_interval_improvement.py
python combine_scheduling_comparison.py

cd 03_RQ2_Sensitivity_Analysis_Data
python plot_rq2_repeat_patterns_horizontal.py
python plot_rq2_repetition_rate_vs_capacity.py
python analyse_rq2_quality_scores.py

cd 04_Memory_Layout_Analysis_Data
python3 analyze_memory_layout_all_models.py

cd 06_RQ4_Baseline_Comparison_Data
python3 generate_rq4_evidence_metrics.py
python3 combine_baseline_comparison.py

If a script requires Python packages such as pandas, numpy, matplotlib, seaborn, or openpyxl, install them in a local virtual environment before running the script.

Running the WASM Examples

The included source code can be rebuilt and executed with local WASM runtimes such as Wasmtime, WAMR, or WAVM. Large model files are not included and must be supplied separately.

A typical run flow is:

1. Place the external model files in local model directories.
2. Build the relevant WASM example from 01_Memoir_Code_and_Artifacts/sources/candle_memoir/ with the desired policy feature.
3. Use the compiled binaries under 01_Memoir_Code_and_Artifacts/wasm_used/.
4. Pass model directories through environment variables such as QWEN3_MODEL_DIR, SMOLLM2_MODEL_DIR, QWEN15_MODEL_DIR, or QWEN2_MODEL_DIR.
5. Enable memory logging with CANDLE_MEMLOG_ENABLE=1 and CANDLE_MEMLOG_FILE=<output-path> when collecting memory logs.

See 01_Memoir_Code_and_Artifacts/README.md for a concrete Wasmtime command template.

Data Integrity and Packaging Notes

This submission package was prepared to contain only the core review artifact. The following classes of content are intentionally excluded:

• Manuscript drafts and private writing folders
• Historical archive folders
• Local helper symlinks and machine-specific workspace links
• Git metadata from the copied source trees
• Large external model weights
• Non-paper figure archives

The package keeps raw experimental outputs where they are needed for traceability. Some raw output files include runtime log strings or model-generated natural language text. These should be interpreted as recorded experiment outputs, not as repository documentation or data-provenance statements.

Publication and Distribution

This section follows common artifact-evaluation practice: GitHub for browsing and collaboration, Zenodo for a citable DOI archive, and Hugging Face only for external model weights (not included here).

What goes to GitHub

Push the entire contents of this directory (Memoir_JSA_Submission_Artifact/) as the repository root. That includes:

Do not push from the parent workspace folder; only this artifact directory.

Suggested release tag: v1.1-jsa-artifact-20260524

Live repository: https://github.com/Rhys-Wang-wannaLearnMath/Memoir_JSA_Submission_Artifact

What goes to Zenodo

Zenodo should archive the same content as the GitHub release (not a superset). Recommended workflow:

1. Connect GitHub to Zenodo and enable archival for this repository.
2. Create a GitHub Release with tag v1.1-jsa-artifact-20260524.
3. Let Zenodo auto-ingest the release tarball/zipball.
4. Fill Zenodo metadata (title, authors, keywords, license Apache-2.0).
5. Copy the Zenodo DOI back into README.md, 00_REVIEWER_INDEX.md, and CITATION.cff.

Zenodo is the canonical citable archive for the paper artifact appendix. Attach CHECKSUMS.txt to the GitHub Release assets for integrity verification.

What goes to Hugging Face

Do not upload this artifact to Hugging Face. HF is for model weights only.

Download the three model families used in experiments from Hugging Face and place them locally. See MODEL_WEIGHTS.md for direct links:

• Qwen1.5-0.5B → Qwen/Qwen1.5-0.5B
• Qwen3-0.6B GGUF → Qwen/Qwen3-0.6B-GGUF
• SmolLM2-360M GGUF → HuggingFaceTB/SmolLM2-360M-Instruct-GGUF

In the paper and README, cite Zenodo DOI for the artifact and Hugging Face model IDs for weights.

What stays local (do not publish)

• Parent workspace files (参考过程.md, manuscript drafts, private notes)
• Cargo target/ build caches (excluded via .gitignore)
• Downloaded model weight directories
• API keys for optional LLM-judge re-runs

Suggested Citation Placeholder

After Zenodo creates the DOI, update here, in CITATION.cff, and in the paper:

Memoir JSA Submission Artifact (v1.1-jsa-artifact-20260524). Zenodo. DOI: <to be filled>

Contact

For artifact questions, use the contact information provided in the corresponding Journal of Systems Architecture submission.