LLM Evaluation Pipelines (Bare / LMQL / RAG)

This repository contains a set of three pipelines to run and compare multiple LLM outputs on a shared question set. It consists of three execution styles:

• Bare: direct transformers generation (model.generate) over prompts.
• LMQL: runs prompts through an LMQL server to support structured and constrained querying.
• RAG: retrieval-augmented generation using a ChromaDB vector store, an embedding model, and a generator model.

Repository layout

code/
  main.py
  model_id_list*.json
  pipelines/
    bare/   # transformers-only evaluation
    lmql/   # LMQL server and worker orchestration
    rag/    # ChromaDB, embeddings, generator model
  utils/

Key files:

• code/main.py: entrypoint, selects a pipeline (bare, lmql, rag).
• code/model_id_list*.json: curated lists of models grouped by organisation
• code/pipelines/*/*_config.json: pipeline configuration (paths, batch sizes, timeouts, etc.).
• code/pipelines/*/*.job: cexample SLURM submission scripts (cluster usage).

Prerequisites

• Python 3.10+ (3.11/3.12 should also work if your ML stack supports it)
• For GPU runs: CUDA-enabled PyTorch
• Disk space: model downloads can be large, scripts attempt to explicitly delete HF caches to reduce disk pressure.

Dependencies (high level)

Bare

• torch
• transformers
• accelerate (recommended)
• bitsandbytes (only if you run bnb-4bit models)

LMQL

• lmql
• plus the Bare dependencies

RAG

• chromadb
• sentence-transformers
• langchain (text splitter)
• unsloth (used by the current RAG runner)
• plus the Bare dependencies

Data layout

The minimum expected structure is:

data/
  in/
    tasks/
      test_task_0.json
      generate_task.json   # or other task_*.json files
    data_ground_truth/
      ...                  # RAG documents (folder of .txt/.md/.json)
  out/
    bare/
    lmql/
    rag/
  chroma_db/               # persistent vector DB (RAG)

Questions file format

Bare and RAG expect a JSON list of objects containing a prompt field, e.g.:

[
  {"prompt": "What is the capital of France?"},
  {"prompt": "Explain cochlear implants in simple terms."}
]

Model list files

Model lists live in code/model_id_list*.json and are grouped by organisation, for example:

{
  "Google": [
    {"model_id": "unsloth/gemma-2b-bnb-4bit", "type": "causal", "...": "..."}
  ],
  "Meta": [
    {"model_id": "unsloth/llama-2-7b-bnb-4bit", "type": "causal", "...": "..."}
  ]
}

The pipelines typically rely on:

• model_id — Hugging Face model identifier
• type — "causal" or "seq2seq"

Running the pipelines

1) Bare pipeline (recommended entrypoint: bare_run1.py)

Outputs:

• JSON results in paths.OUTPUT_DIR (timestamped filenames)
• Checkpoints in paths.OUTPUT_DIR/checkpoints/
• Logs default to data/out/bare/logs/bare_pipeline.log

Result format:

{
  "model_id": "…",
  "timestamp_utc": "YYYYMMDD_HHMMSS",
  "results": [{"question": "...", "answer": "..."}]
}

2) LMQL pipeline

• LMQL starts a server per model and restarts it between chunks (to improve stability).
• Outputs are written into per-model timestamped directories under paths.OUTPUT_DIR.
• A merged JSON is written at the end of each model run.

3) RAG pipeline

• Per-GPU JSONL shards (if multi-GPU) and a merged JSON output in the configured output directory.

Troubleshooting and Housekeeping

• Out-of-memory (GPU): lower BATCH_SIZE and/or MAX_LENGTH in the pipeline config.

• 4-bit models fail on CPU: bnb-4bit models require CUDA + bitsandbytes.

• Disk pressure from model downloads: some runners delete per-model cache directories after runs, confirm HF cache location and available quota.

• Gated models: export HF_TOKEN / HUGGINGFACE_HUB_TOKEN or run huggingface-cli login.

• Use code/utils/add_tokenizer.py to add/check tokenizer fields in the model list.

• Use code/utils/filter_missing_models.py to produce filtered model lists by substring matching.

Model licenses vary by model and are listed (when available) in the model list JSON files. Ensure you comply with the relevant model terms.