Pixelated Empathy AI - Production Structure

Overview

Production-ready AI system for empathy-aware conversational AI with proper organization and deployment structure.

Directory Structure

ai/
├── training/                    # Model training, fine-tuning, safety, judges
│   ├── configs/                # Hyperparameters, infra, model/stage configs
│   ├── scripts/                # Training scripts (gpu, services, sprint8)
│   ├── tests/                  # Training test suite
│   ├── rlhf/                   # RLHF / DPO / GRPO trainers
│   ├── sdg/                    # Synthetic data generation
│   ├── defense_mechanisms/     # Defense mechanism training
│   ├── coaching_safety/        # Coaching safety modules
│   ├── sliced/                 # Stage-sliced datasets (stage1_foundation, stage2_*)
│   ├── utils/                  # Shared training utilities
│   ├── checkpoints/            # Model checkpoints
│   └── *.py                    # Flat trainers/judges/scorers (see note below)
├── data/                       # Datasets and data management
│   ├── raw/                    # Raw datasets
│   ├── processed/              # Processed datasets
│   ├── curated/                # Curated datasets (DVC-tracked)
│   └── synthetic/              # Synthetic data generation
├── benchmarks/                # Benchmarks
│   ├── *_performance_baseline.json  # CPU/memory/inference benchmark baselines
│   └── tests/                 # Benchmark tests
├── models/                     # Model artifacts and exports
│   ├── checkpoints/            # Saved model checkpoints
│   ├── artifacts/              # Model artifacts
│   ├── exports/                # Exported models for deployment
│   ├── base/                   # Base model definitions
│   ├── moe/                    # Mixture-of-experts
│   └── pixel/                  # Pixel model
├── inference/                  # Deployment and inference
│   ├── api/                    # API endpoints (cms, mcp_server, memory, techdeck)
│   ├── services/               # Inference services
│   └── deployment/             # Deployment configs (helm, k8s, s3, postgres)
├── pipelines/                  # Data and model pipelines
│   ├── data_processing/        # Data processing (academic, journal, youtube, extractors)
│   ├── model_training/         # Training pipelines
│   ├── evaluation/             # Model evaluation
│   ├── orchestration/          # Stage organizers
│   ├── voice/                  # Voice / audio pipelines
│   └── edge_case/              # Edge case generation
├── sourcing/                   # Data sourcing (academic, journal, youtube)
├── research/                   # Research and experimentation
│   ├── notebooks/              # Jupyter notebooks
│   ├── experiments/            # Research experiments
│   ├── analysis/               # Analysis and reports
│   ├── gates/                  # Inference gates (consent, crisis, pii)
│   └── reflection/             # Reflection memory modules
├── tools/                        # Utilities and tools
│   ├── utilities/              # Utility modules (api, core, data, pipelines, pkg_mera)
│   ├── scripts/                # Shell scripts and command-line tools
│   ├── generators/             # Data generators
│   └── DataDesigner/           # External NeMo Data Designer snapshot
├── qa/                         # Quality assurance
│   ├── reports/                # QA reports
│   ├── validation/             # Validation (crisis_detection, diagnosis_arena, therapy_bench)
│   └── testing/                # Test suites
│       └── legacy_tests/       # Archived broken legacy tests
├── annotation/                 # Annotation agents and API
├── compliance/                 # Compliance (db, security, validators)
├── prompts/                    # Prompt templates (agents, clinical, safety, system)
├── configs/                    # Config management (envs, models, monitoring, legacy)
├── docs/                       # Documentation (api, architecture, guides, ops)
├── scripts/                    # Maintenance scripts
├── migrations/                 # DB migrations (SQL)
├── experiments/                # Experiment scratch space
├── assets/                     # Static assets
└── ops/                        # Ops tooling (Dockerfile, Makefile, CI)

Note on flat training/*.py: ~70 modules (trainers, judges, scorers, SDG pipelines) live at the training/ root and are heavily cross-imported (training.clinical_validity_scorer, training.pixelated_production_pilot, training.dpo_trainer, etc.). They are importable library code, not runnable scripts, and remain flat by design until a dedicated package split is done.

Quick Start

Environment

# Sync dependencies (Python 3.13 via uv)
uv sync

Run Tests

uv run pytest

Training

uv run python -m training.<entry_point>

Inference

uv run python -m inference.api.<module>

Test Suite & Training Pipeline

Python Test Suite

Run all training tests (excluding legacy book PDF converter tests):

uv run pytest training/tests/ --ignore=training/tests/test_book_pdf_converter.py -q

Coverage

Coverage is enforced by CI (.github/workflows/training-safety-coverage.yml) at two tiers:

• Safety-critical (95 % aggregate, branch coverage): training.shared_config, training.multilingual_safety_checker, training.clinical_safety_checker, and training.reward_score. These four modules gate the clinical decision logic — the 95 % threshold accounts for two unreachable ModuleNotFoundError fallback lines in multilingual_safety_checker.py that cannot be exercised under the editable install used in CI.

• Pilot module (40 %): training.pixelated_production_pilot. The pilot module is a 1,043-line production SFT pipeline whose body resolves at runtime only with the full transformers + trl + peft stack on GPU. Unit tests deliberately cover only the dataclass/CLI/path-safety surface (config, args, HubConfig, RunConfig, _maybe_push_to_hub, safe_path, CheckpointVerificationCallback). ML-pipeline correctness is validated separately via the training runnable and smoke tests.

Run a tier locally (matches what CI runs):

# Safety-critical gate
uv run pytest training/tests/ --ignore=training/tests/test_book_pdf_converter.py \
  --cov=training.shared_config \
  --cov=training.multilingual_safety_checker \
  --cov=training.clinical_safety_checker \
  --cov=training.reward_score \
  --cov-branch --cov-fail-under=95 -q \
  --cov-report=xml:coverage/safety-critical-coverage.xml \
  --cov-report=term

# Pilot gate
uv run pytest training/tests/ --ignore=training/tests/test_book_pdf_converter.py \
  --cov=training.pixelated_production_pilot \
  --cov-branch --cov-fail-under=40 -q \
  --cov-report=xml:coverage/pilot-coverage.xml \
  --cov-report=term

---

Notes on the previous 100 % safety-critical threshold

Earlier revisions of this README and the workflow targeted 100 % coverage across the safety-critical modules. That gate failed in practice because two ModuleNotFoundError fallback lines in multilingual_safety_checker.py (the absolute from ai.training.clinical_safety_checker import ... fallback) are unreachable under an editable install — Python's import machinery always resolves the relative from .clinical_safety_checker import first when the training package is on sys.path. The fall-through is a defensive idiom for distribution layouts where training is not a discoverable package; it is not a regression target. The 95 % gate documents this honestly.

If the 100 % threshold is required for regulatory reasons in the future, the test test_fallback_import_path_used will need a custom sys.meta_path finder to force the relative import to fail deterministically. That's tracked as a follow-up; do not silently lower the threshold from 95 % to 100 % again.

DPO / GRPO Smoke Tests

Lightweight smoke tests for direct preference optimization and group-relative policy optimization:

uv run pytest training/tests/test_dpo_trainer.py training/tests/test_grpo_trainer.py -q

Training Pipeline Environment

The training pipeline runs on a CPU-only PyTorch stack. The [tool.uv.sources] section in pyproject.toml pins torch, torchvision, and torchaudio to the pytorch-cpu index to avoid pulling CUDA binaries that would fail on CPU-only runners.

Safety Checker Environment Variables

Set the following environment variable in test/CI environments to disable ML model loading for safety checkers (enabled by default in conftest.py):

export AI_DISABLE_SAFETY_ML_MODELS=1

Production Deployment

Configuration

• Environment configs: configs/envs/{development,staging,production,testing}/
• Model configs: configs/models/
• Monitoring configs: configs/monitoring/

Model Management

• Training checkpoints: training/checkpoints/
• Model artifacts: models/artifacts/
• Exported models: models/exports/

Data Management

• Raw datasets: data/raw/
• Processed datasets: data/processed/
• Curated datasets: data/curated/ (DVC-tracked)
• Synthetic data: data/synthetic/

API Deployment

• API endpoints: inference/api/
• Deployment configs: inference/deployment/
• Service implementations: inference/services/

Development Workflow

1. Data Preparation: Use pipelines under pipelines/data_processing/
2. Data Sourcing: Use sourcing/
3. Model Training: Use training/ (flat modules + training/scripts/)
4. Model Evaluation: Use pipelines/evaluation/
5. Deployment: Use configurations in inference/deployment/
6. Quality Assurance: Use qa/validation/ and qa/testing/

Legacy Files

All previous task files and reports are kept in their functional homes (qa/reports/ for PR/audit data, tools/DataDesigner/ for the external Data Designer snapshot, qa/testing/legacy_tests/ for broken legacy tests).

Environment Setup

# Sync dependencies via uv
uv sync

# Activate virtual environment
source .venv/bin/activate

Support

• Documentation: docs/
• API Reference: docs/api/
• Architecture Guide: docs/architecture/