SafeWatch + Nemotron Nano V2 VL (local workflow)

This document describes fork/local additions for training and evaluating a video guardrail model on SafeWatch-style SFT JSONL (intervals + multi-label categories C1–C6), using the Nemotron Nano V2 VL LoRA recipe in Megatron-Bridge.

---

1. Dataset: make_safewatch_dataset

Location: src/megatron/bridge/data/vlm_datasets/hf_dataset_makers.py (make_safewatch_dataset)

Role: Load SafeWatch SFT JSONL (e.g. sft_train_v2.jsonl / sft_eval_v2.jsonl) into conversation examples the Nemotron VL collate path expects: user turns include {"type":"video","path": "<mp4>"} and assistant targets carry structured JSON (shots, intervals, guardrails).

Typical YAML (examples/recipes/nemotron_vl/conf/nemotron_nano_v2_vl_safewatch.yaml):

dataset:
  do_test: false   # see §2 — avoids duplicate eval on the same eval JSONL
  sequence_length: 8192
  maker_name: make_safewatch_dataset
  maker_kwargs:
    train_jsonl_path: /path/to/sft_train_v2.jsonl
    eval_jsonl_path: /path/to/sft_eval_v2.jsonl
    max_examples: 100
    require_video_exists: true

Provider wiring: HFDatasetConversationProvider in src/megatron/bridge/data/vlm_datasets/hf_provider.py registers make_safewatch_dataset and builds train / validation / test splits from the maker. SafeWatch maps both validation and test splits to the same eval_jsonl_path; without do_test: false, the training driver would run two full eval passes (validation + test) on the same data.

---

2. Training stack behavior (why we changed a few files)

---

3. Shell scripts (roles)

train.sh

• Runs LoRA fine-tuning via examples/recipes/nemotron_vl/finetune_nemotron_nano_v2_vl.py with nemotron_nano_v2_vl_safewatch.yaml.
• Loads base weights from --pretrained-checkpoint (e.g. Megatron Nemotron VL checkpoint dir).
• Uses torchrun --nproc-per-node=... for multi-GPU data parallel.
• Optional: set WANDB_API_KEY in the environment (do not commit secrets); YAML/logger overrides for project name can be passed on the CLI.

inference.sh

• No training: train.skip_train=true.
• Loads a fine-tuned run with checkpoint.load=<results_dir> and checkpoint.ckpt_step=<step> (e.g. step 100 → iter_0000100 checkpoint layout under that directory).
• Still uses --pretrained-checkpoint for architecture / tokenizer binding.
• Overrides logger.eval_gt_jsonl_path and logger.eval_dump_dir for dump location.
• Produces results/.../eval_json_full/iter_<step>/records.jsonl (one JSON object per line).

eval.sh

• Offline metrics only (CPU-friendly): runs scripts/eval_interval_guardrail_metrics.py on one or more eval_dump roots (directories containing iter_*/records.jsonl).
• Aggregates tIoU (greedy match at τ), guardrail F1 (video-mean vs chunk-pool vs per-category), accuracy, TN counts, etc. See script docstring and §5.

---

4. Output layout

<eval_dump_dir>/
  iter_0000100/
    records.jsonl    # one record per line: gt intervals, pred intervals, optional dump fields

Legacy flat iter_*_pred_*.json / iter_*/pred_*.json are still supported by the metrics script when records.jsonl is absent.

---

5. Metrics script (scripts/eval_interval_guardrail_metrics.py)

Inputs: --dirs (one or more roots), --tiou-threshold or --tiou-thresholds, --unmatched-policy (strict | matched_only).

Notable outputs:

• Video-level headline F1: mean of per-video micro-F1 (after pooling label bits within each video across matched/unmatched chunks).
• Chunk-pool: single global confusion over all binary guardrail decisions (TP/FP/FN/TN); micro F1 uses only TP/FP/FN (TN excluded by definition); accuracy = (TP+TN)/(TP+TN+FP+FN).
• Per-category: binary F1 and accuracy per guardrail key (e.g. C1–C6).
• chunks= / bits=: guardrail rows (segment-level vector evals) vs total binary decisions.

Optional: --csv, --json-out.

---

6. Example: metrics console (illustrative)

After running eval.sh or the Python command directly, you may see blocks similar to:

=== tIoU>=0.5  unmatched=strict ===
100 | f1=0.789 f1_all=0.752 tiou=0.860 cov=0.404 hit=0.44 n_f1=14/32

--- chunk_pool: counts, TN, accuracy, F1 ---
100 | chunks=83 bits=498 acc=0.970 | pool_f1=0.516 P=0.421 R=0.667 | tp=8 fp=11 fn=4 tn=475 | f1: C1(...)=1.000 ... | acc: C1(...)=1.000 ...

Numbers depend on dataset size, τ, checkpoint, and whether eval_dump used a subset (eval_dump_max_records).

---

7. Training length vs epochs (rule of thumb)

With train_iters optimizer steps and global_batch_size samples per step:

[ \text{approx epochs} \approx \frac{\texttt{train_iters} \times \texttt{global_batch_size}}{N} ]

where (N) is the number of unique training rows after caps/filters (e.g. max_examples and require_video_exists).

---

8. Entry points (code map)