alice-ingest — Cardboard Airplane

 █████╗ ██╗     ██╗ ██████╗███████╗
██╔══██╗██║     ██║██╔════╝██╔════╝
███████║██║     ██║██║     █████╗
██╔══██║██║     ██║██║     ██╔══╝
██║  ██║███████╗██║╚██████╗███████╗
╚═╝  ╚═╝╚══════╝╚═╝ ╚═════╝╚══════╝
        A Large Ion Collider Experiment

A recreation of the ALICE O2 Scalable Logging Architecture, built at two scales:

• Paper airplane — the whole pipeline shrunk onto one machine with Docker Compose, for hands-on learning and failure experiments.
• Cardboard airplane — the same pipeline as an actual distributed system: 5 CERN OpenStack VMs, native systemd services (no Docker), one 5-node two-tier OpenSearch cluster (2 worker + 3 storage), provisioned and configured end-to-end with Ansible.

Both replay real CERN S3 logs (from the epn-backup-logs bucket — or mock logs offline) through Fluent Bit → OpenSearch → Dashboards, with index patterns auto-provisioned on startup. Still no Kafka, no Grafana (next flight).

The cardboard airplane is the jump from paper to cardboard: from "Docker Compose on one machine" to real multi-VM provisioning and a genuine OpenSearch cluster. v2 split that cluster into two tiers — 2 worker nodes holding disposable, node-local info logs, plus 3 replicated storage nodes for the valuable other + infologger logs. v3 added a triggered replay button (make replay), more real data, and the auto-provisioned ALICE Cockpit dashboard. v4 (this tree) migrates the native stack to OpenSearch + Dashboards 3.7 (fixing saved-search restore) and finishes the cockpit with a platform-health band: cluster + per-index state, Fluent Bit per node, and Dashboards self-health. The single-machine path is unchanged and remains the local dev loop.

Design target — the real platform we simplify from: docs/ARCHITECTURE.md.

---

Cardboard airplane — distributed deploy (5 VMs, two-tier, native, no Docker)

Provisions 5 CERN VMs and configures a 5-node two-tier OpenSearch cluster (2 worker

• 3 storage) + worker Fluent Bit + S3-replay producers, with Dashboards (nginx TLS + basic-auth) on one storage control node. Full runbook — topology, the two auth paths, rolling-safety, verification — is deploy/README.md; the essentials:

Prereqs.

1. OpenStack auth — on lxplus, kinit plus the six OS_* exports (v3fedkerb); see deploy/README.md §3.1.
2. Control-node toolchain — make bootstrap builds a self-contained .venv with ansible-core, openstacksdk, keystoneauth1[kerberos] (needed for v3fedkerb), and the python-openstackclient CLI, plus the Galaxy collections. The deploy targets use it automatically; see deploy/README.md §3.2.
3. Secrets (vault) — the [cern_s3] keys and the Dashboards basic-auth password live in an encrypted vault, never plaintext:

   cd deploy
   cp group_vars/vault.yml.example group_vars/vault.yml
   $EDITOR group_vars/vault.yml            # fill in the 3 real values
   ansible-vault encrypt group_vars/vault.yml
   

Clean-slate prerequisite. v2 assumes the OpenStack project is empty or already running v2. It does not know about v1 and won't clean it up. Because it reuses the VM names alice-ingest-1..5, a still-running v1 stack (alice-ingest-1..3) would be silently adopted into a broken mixed cluster — tear v1 down first (ansible-playbook teardown.yml from a cardboard-airplane-v1 checkout). See deploy/README.md §3.

Fly (from the repo root):

make provision     # create the 5 OpenStack VMs (idempotent) — needs OpenStack auth
make deploy        # configure the cluster — prompts for the vault password
make replay        # load real logs (the "replay button" — no vault needed)
make poison        # background calibration of every one-minute detector

make deploy arms the pipeline but ingests nothing; make replay is the deliberate load step (POSTs /replay to each worker). Because replay has no dedup, use make replay-fresh to wipe and reload cleanly rather than a second make replay. See docs/CARDBOARD-AIRPLANE-V3.md.

make poison (alias: make poison-replay) starts or continues that paced baseline, waits until all ten one-minute RCF detectors are actually trained, and then injects labelled outlier windows into entities already modelled by the live replay. It follows each injection through native AD results, projected incident episodes, and seven deterministic monitor paths. The seven 30-minute detectors are excluded. Use make poison-status to poll it and make poison-stop to cancel it. The original misspelling make posion-replay is accepted as an alias.

View. Dashboards on the control VM: https://<control-VM>:5601, user alice (the vault password), self-signed cert. From outside CERN, tunnel through lxplus:

ssh -L 5601:<control-VM-internal-ip>:5601 lxplus.cern.ch    # then https://localhost:5601

Open the ALICE Cockpit dashboard (auto-provisioned) for the unified view — logs on top, platform health (cluster, per-index, Fluent Bit per node, Dashboards itself) below — or Discover on the default infologger,application-logs-* pattern with the seven seed saved searches (which apply their query on open, the v4 fix). For a browser-driven load without the CLI, the control node also serves an ops page at https://<control-VM>:5601/ops (same basic-auth) with safe post/redirect/get actions for Reload data (fresh) / Append replay / Stop replay / Clear findings / Poison replay / Stop poison, live button progress, and live detection counts. Refreshing an action result never submits the action again. Data is historical (~June 2026, pinned by RUN_TAG) — if Discover looks empty, widen the time range rather than assuming no data.

Teardown.

make teardown      # delete the 5 VMs, re-close 5601, drop the generated inventory

---

Paper airplane — single machine (Docker Compose)

Prereqs: Docker + Docker Compose v2, and a Docker VM with real resources — OpenSearch alone wants a 3 GB heap (on macOS/Colima: colima start --cpu 8 --memory 32 --disk 100).

1. CERN S3 credentials (one-time). The replay container reads real ALICE logs using an AWS profile named cern_s3. Put your keys in ~/.aws/credentials on the host — Compose mounts this file read-only into the replay container only:

# ~/.aws/credentials
[cern_s3]
aws_access_key_id     = <YOUR_CERN_S3_ACCESS_KEY>
aws_secret_access_key = <YOUR_CERN_S3_SECRET_KEY>

2. Launch.

make run

Brings up OpenSearch + Dashboards + one node + the replay service. Replay autostarts and streams real DDS / stdout / InfoLogger logs through Fluent Bit into OpenSearch. First boot: OpenSearch takes ~30–90 s to go healthy (longer under CPU contention — it's booting, not stuck).

3. View. Open http://localhost:5601 → Discover. The three index patterns (infologger, application-logs-local, application-logs-central) are already created — no manual setup. Pick one, set a time range, and browse.

Teardown.

make down            # stop, keep data volumes
make down volume     # stop + wipe all volumes (fresh next run)

No CERN credentials? Run fully offline with mock producers instead:

make mocks           # same pipeline + auto-patterns, synthetic logs, no S3