Vedaz AI Engineer assessment — Hardik Tiwari

Submission for the AI Engineer (AI Astrologers) assessment. It covers both stages plus the final round.

Final round (in final/)

• vedaz_finetune_qwen.ipynb — fine-tunes Qwen2.5-1.5B-Instruct (LoRA, 4-bit, free Colab T4) on the provided chats, shows a before/after, then merges the adapter and exports/pushes the model for serving.
• vllm_hosting.md — the write-up: hosting the fine-tuned model on a GPU VPS with vLLM as an OpenAI-compatible API (driver, install, serve, systemd, nginx/TLS, scaling).
• data/vedaz_chats.jsonl — the provided 55 chats, curated with my Stage 2 checker: 5 exact duplicates removed → 50 unique clean chats (details in the notebook).

The three rounds connect end to end: the Stage 2 checker curated the final round's training data, and the merged model from the notebook is exactly what the vLLM guide serves.

Repo structure

stage1/
  stage1_review.md          Task 1: my review of the 15 example chats
  vedaz_new_chats.jsonl      Task 2: 5 new example chats I wrote (id + tags + messages)
  build_chats.mjs            the small script that emits the jsonl (for clean escaping)
stage2/
  checker.py                 Task 1: structure + length + duplicate + split + safety checker
  generator.py               Task 2: AI chat generator that self-filters with the checker
  quality_tester.py          Task 3: LLM-as-judge quality + safety tester
  llm.py                     shared, provider-agnostic LLM helper
  data/                      provided dataset + self-test files (see "Data files" below)
  outputs/                   generated chats, quality report, train/test split
  requirements.txt
  .env.example
bonus/
  vedaz_lora_finetune.ipynb  optional LoRA fine-tune notebook (Colab)

Setup (Stage 2)

cd stage2
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env        # then fill in your key

The scripts speak the OpenAI-compatible API, so they work with OpenAI, Together AI, or DeepSeek by setting two env vars (see .env.example). Keys are read from the environment, never hardcoded.

checker.py runs with no API key and no dependencies (it uses a keyword safety layer by default). generator.py and quality_tester.py need a key.

Stage 2 — how to run

# 1. Checker: report + train/test split on the provided data
python checker.py data/vedaz_astrologer_finetune.jsonl --split

# (proof the safety detector actually catches violations, not just passes everything)
python checker.py data/unsafe_examples.jsonl

# 2. Generator: make 10 clean chats (auto-filtered through the checker)
python generator.py --n 10 --out outputs/generated_chats.jsonl

# 3. Quality tester: grade answers for safety + warmth + honesty
python quality_tester.py --out outputs/quality_report.md

# optional: add the LLM safety layer to the checker
python checker.py data/vedaz_astrologer_finetune.jsonl --llm

Task 1 — the checker

Reads a .jsonl and reports: structure (system first, then alternating user/assistant, non-empty), length (words + rough tokens), exact and near-duplicates (normalized text similarity), a deterministic train/test split, and safety flags.

Safety detection method and its blind spots (the important part):

I use two layers.

1. Keyword/regex (default, free, fast, tuned for high recall). It scans only the assistant turns (those are what the model produces and what we control) and flags danger using outcome-anchored patterns plus co-occurrence windows — an assertive word ("zaroor", "100%", "guaranteed", "pakka", "nishchit") within a few words of an outcome word ("naukri", "beta", "paisa", "theek ho"), in either order — across Hindi, Hinglish, and English. It covers the three rules (predicting death/serious illness, guaranteeing a money/medical/marriage/child outcome, fear-selling a paid remedy) and adds a light check for the assistant endorsing a dangerous claim the user just made. A match is suppressed only when a tight, specific refusal phrase sits right next to it — "won't predict", "no guarantee", "गारंटी नहीं", "भविष्यवाणी नहीं", "ज़रूरी नहीं", or a clear scam-refutation ("jhooti guarantee", "galat hai"). Generic words like "doctor" or "consult" are deliberately not suppressors: an earlier version treated them as such and it silently hid a real death prediction ("…mrityu nishchit hai" sitting near "consult"). A safety tool should fail loud. On the provided 15 it produces zero false positives; on a deliberately-unsafe file it catches all three violation types; and a regression file of 12 realistic violations + 3 safe look-alikes (data/safety_regression.jsonl) all classify correctly.

   Where it misses (stated plainly): the keyword layer catches only a fraction of the ways an unsafe astrologer could phrase a death prediction or a guarantee — novel wording, heavy spelling variation, sarcasm, or a violation split across two far-apart sentences will slip past it. For the death rule especially, treat the keyword layer as a tripwire and the LLM layer as the real net. Two specific limits worth naming: (a) "Cancer" is also the Kark zodiac sign, so the disease pattern only fires when a medical-context word is nearby and explicitly skips the rashi/zodiac sense — a determined edge case could still evade it; (b) scanning only assistant turns means an assistant that agrees with a dangerous user message without restating the trigger words ("haan, woh sahi keh rahe hain") relies on the endorsement check or the LLM layer to be caught. Keywords are a cheap first pass, not a final judge.

2. LLM classifier (optional, --llm). Sends each assistant turn — together with the preceding user turn for context — to a model that judges it against the four rules and returns JSON. This catches the phrasing the keywords miss and the endorsement-by-reference case, at the cost of API calls and latency.

A real pipeline at scale runs the cheap keyword pass over everything, then the LLM pass over the flagged and a random sample, which keeps cost down while catching what keywords cannot.

Data files

stage2/data/ holds:

• vedaz_astrologer_finetune.jsonl — the dataset in JSONL (one chat per line). All the tooling consumes this file.
• vedaz_astrologer_finetune.json — the raw provided file (a pretty-printed JSON array). Kept for reference only; the checker auto-detects an array and parses it, but the .jsonl is canonical.
• unsafe_examples.jsonl — 3 deliberately-unsafe chats, to prove the detector flags rather than just passes everything.
• safety_regression.jsonl — 12 realistic violations (the kinds that an earlier version of the detector missed) plus 3 safe look-alikes (the "Cancer" zodiac sign, a real no-guarantee disclaimer, a genuine doctor redirect). Running the checker on it shows 12 flagged and 3 clean — the regression guard for the safety logic.

Task 2 — the generator

Asks an LLM to write a chat for a given situation, returns strict JSON, prepends one canonical system prompt (I inject it rather than trusting the model, which guarantees a consistent safety contract across all generated data, a problem I flagged in Stage 1), then runs each chat through the Task 1 checker and keeps only the ones that are well-formed and safety-clean. It over-generates and filters until it has enough. The seed topics deliberately include cases the original 15 were missing (crisis, jailbreak/pressure, grief, low-effort, fertility), so the generated data widens coverage.

Task 3 — the quality tester

Sends a mix of everyday and high-risk test questions (crisis, health, money-guarantee bait, a fear-selling trap) to the assistant under test, then grades each answer with a second model acting as judge, scoring safety, warmth, and honesty (1-5) plus a hard safe_pass gate. Prints a table and writes a Markdown report. Using a separate judge model gives a repeatable score instead of a gut feel, and the judge is explicitly told that refusing or redirecting is the correct, high-safety behavior.

Optional bonus — LoRA fine-tune

bonus/vedaz_lora_finetune.ipynb is a Colab notebook that LoRA-fine-tunes a small open model on the example chats and shows a before/after generation. It is set up for a free Colab GPU. This is the bonus, included for completeness.

Notes

• I used AI tools to help write this (which is the job), and I own and can explain every line.
• The biggest gap I found in the data, and the thing I would prioritise next, is the total absence of mental-health-crisis examples. My Stage 1 chats and the generator topics both add that case, with a helpline redirect.
• With more time I would: add a proper test suite for the checker, expand the safety patterns with a labelled eval set to measure precision/recall, and make the generator do multi-turn adversarial chats (a user who pushes back after a refusal).