Master's Thesis — Code

The framework measures how novel LLM-generated text is relative to pretraining data, using a three-stage pipeline: dense retrieval (FAISS + GIST embeddings), late-interaction reranking (ColBERTv2), and human baseline calibration.

Models studied: SmolLM and SmolLM2 (360M and 1.7B parameters, base and instruct variants).

Repository Structure

thesis-repository/
│
├── download_data_code/                  Download pretraining corpora + benchmarks
├── dolma/                               HuggingFace dataset loader for Dolma
├── calc_chunks_and_embeddings_code/     Chunk corpora + compute GIST embeddings
├── create_FAISS_index_SmolLM.py/.sbatch Build FAISS HNSW index (SmolLM corpora)
├── create_FAISS_index_SmolLM2.py/.sbatch Build FAISS HNSW index (SmolLM2 corpora)
├── analyze_generations_novelty_code/    Core novelty analysis pipeline
├── effect_of_chunking_borders/          Chunking border analysis experiment
├── eval_benchmarks/                     Model evaluation (lm-eval-harness + custom)
├── eval_benchmarks_lighteval/           GSM8K evaluation (lighteval)
├── results/generation_novelty/          Plotting + statistical analysis
├── sum_cpu_time.py                      Computational cost (Appendix A.5)
├── container.def                        Singularity/Apptainer container definition
└── requirements.txt                     Python dependencies

Pipeline Overview

The full pipeline runs on an HPC cluster using SLURM and Singularity containers. Each stage has Python scripts and corresponding .sbatch job files.

Stage 1: Data Download

download_data_code/ — Download pretraining corpora and benchmark datasets.

dolma/dolma.py — HuggingFace dataset loader for Dolma.

Stage 2: Chunking + Embeddings

calc_chunks_and_embeddings_code/ — Split corpora into fixed-size token windows and compute GIST-large embeddings.

Datasets chunked: Cosmopedia v2, DCLM, FineMath, FineWeb-edu, InfiWebMath, Python-edu, Stack-edu, GSM8K, OpenRewrite, TruthfulQA, instruct data, OpenHermes-2.5.

Stage 3: FAISS Index Construction

Stage 4: Novelty Analysis

analyze_generations_novelty_code/ — Core novelty analysis pipeline.

Stage 5: Model Evaluation

eval_benchmarks/ — Model evaluation using lm-eval-harness and custom evaluations.

eval_benchmarks_lighteval/ — GSM8K evaluation using lighteval.

Stage 6: Chunking Border Experiment

effect_of_chunking_borders/ — Analyzes how chunking borders affect retrieval results (Figure A.2).

Stage 7: Plotting + Statistical Analysis

results/generation_novelty/ — All plotting and statistical testing scripts.

Miscellaneous

Infrastructure

Container

The Singularity/Apptainer container is defined in container.def:

• Base image: Ubuntu 24.04
• Python 3.11 (via uv)
• PyTorch with CUDA 12.8
• All dependencies from requirements.txt

Build with:

apptainer build container.sif container.def

Running Jobs

All scripts are designed to run inside the container via SLURM. The general pattern:

singularity run \
    --bind $(pwd):/mnt \
    --bind $HF_HOME:/hf-cache \
    container.sif \
    bash -c "export HF_HOME=/hf-cache && python3 /mnt/script.py --job-num $SLURM_ARRAY_TASK_ID"

Data Paths

All scripts run inside a Singularity container with bind mounts:

Environment Variables

Key Dependencies

• FAISS (faiss-cpu): Dense retrieval with HNSW indices
• sentence-transformers: GIST-large embedding model
• pylate / ragatouille: ColBERTv2 reranking
• datasets / huggingface-hub: Data loading and publishing
• spacy (en_core_web_sm, en_core_web_trf): Text processing
• scipy: Wilcoxon rank-sum tests
• matplotlib / pandas / numpy: Plotting and analysis
• sentry-sdk: Error tracking (optional)
• boto3: S3 access for Software Heritage downloads

Thesis Figure/Table → Script Mapping