Paraphrase Investigations

The Limit of LLMs with paraphrased style difference

setup:

cd /scratch_root/ifc24 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh -b -p /scratch_root/ifc24/conda source /scratch_root/ifc24/conda/etc/profile.d/conda.sh unset PYTHONPATH conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r conda create -y -p /scratch_root/ifc24/envs/paraphrx python=3.12 httpx tqdm conda activate /scratch_root/ifc24/envs/paraphrx

reuse it now: source /scratch_root/ifc24/conda/etc/profile.d/conda.sh conda activate /scratch_root/ifc24/envs/paraphrx

Sub Project 1: How do you want to talk to an LLM to have the best outcome

• Development big comparison dashboard
• Flagging style
• people insert something -> ranked how good result -> translate from their style to best style

Data

Initial data from: https://huggingface.co/datasets/tatsu-lab/alpaca

Versions

Too much content - moved over to categories.md!

Generating different versions with gemini-2.5

Test file: politeness, with 5 levels

Inference Checks

Generating the output with for every type of instructions:

python c_assess_inf/src/run_inference.py \
       b_tests/data/alpaca_10_politeness.json \
       c_assess_inf/output/results_test.json \
       --model google/gemma-2b-it \
       --temperature 0 \
       --max_tokens 256

Evaluating each paraphrase answer with Gemini-2.5 using schema:

{
  "is_correct": boolean,        // Did the answer fully satisfy the instruction?
  "score_0_to_5": integer,      // 0 = useless, 5 = perfect
  "explanation": string         // 1-2 sentence rationale
}

-> Produces one JSON object per record whose keys follow the pattern <PARAPHRASE_KEY>_eval.

Metrics

1. Task Fulfilment / Relevance - Does it respond to every part of the prompt? Did it wander off-topic or over-answer?
2. Usefulness & Actionability - Does it translate abstract ideas into concrete advice, examples, or next steps?
3. Factual Accuracy & Verifiabiliy - Are the statements factually correct (no hallucinations)? If it cites sources or internal steps, do those match the final claims?
4. Efficiency / Depth & Completeness - Does it avoid unnecessary verbosity or excessive brevity? Does it cover the key angles, edge-cases, and typical follow-ups? Could the user act on it without having to ask “what about X?”
5. Reasoning Quality / Transparency - Are the steps implicitly or explicitly sound? If uncertain, does it flag that uncertainty instead of bluffing?
6. Tone & Likeability - Is the style friendly and respectful, matching the user’s vibe? Would you enjoy a longer conversation in this voice?
7. Adaptation to Context - Does it use any relevant info the user has shared (location, preferences, prior messages) appropriately?
8. Safety & Bias Avoidance - Does it steer clear of harmful or disallowed content? Does it acknowledge and mitigate possible bias?
9. Structure & Formatting & UX Extras - Is the writing logically ordered and easy to skim? Are lists, code blocks, tables, or rich widgets used when, but only when they genuinely improve readability or utility?
10. Creativity - Does the answer make clever, non-obvious connections that you wouldn’t get from a quick Google search? Or, does it remix ideas, metaphors, or examples in a fresh way rather than serving boilerplate?

'vast' fast version:

sudo apt-get update -qq && sudo apt-get install -y build-essential git
python -m pip install "transformers[torch]==4.42.1" accelerate tqdm
python -m pip install bitsandbytes==0.43.0 flash-attn --no-build-isolation
pip uninstall -y transformer-lens
mkdir logs
mkdir -p c_assess_inf/output/alpaca_prxed/gemma-2-9b-it

WITH FLASH ATTN

nohup python c_assess_inf/src/inference_run_sped_3.py \
        a_data/alpaca/slice_500/context.json \
        c_assess_inf/output/alpaca_prxed/gemma-2-9b-it/context.json \
        --model google/gemma-2-9b-it \
        --batch 128 \
        --max_tokens 128 \
        --temperature 0 \
        --device auto \
        --quant 4bit \
        --log_every 200 \
        --type context \
        >"$LOGFILE" 2>&1 < /dev/null &

WITHOUT

LOGFILE="logs/context_$(date +%F_%H-%M-%S).log"
nohup env DISABLE_FLASH_ATTN=1 HF_TOKEN=HF_TOKEN \
        python c_assess_inf/src/inference_run_sped_3.py \
        a_data/alpaca/slice_500/context.json \
        c_assess_inf/output/alpaca_prxed/gemma-2-9b-it/context.json \
        --model google/gemma-2-9b-it \
        --batch 128 \
        --max_tokens 128 \
        --temperature 0 \
        --device auto \
        --quant 4bit \
        --log_every 50 \
        --type context \
        >"$LOGFILE" 2>&1 < /dev/null &