Prometheus Slurm Exporter 🚀

Prometheus collector and exporter for metrics extracted from the Slurm workload manager — exposes node, partition, job, CPU, GPU, scheduler, fairshare, reservation, and license data, with ten ready-to-use Grafana dashboards and a starter set of site-neutral alerting rules.

[!NOTE] Looking for a next-generation Slurm exporter with native OpenMetrics support (Slurm 25.11+)? Check out my new project: sckyzo/slurm_prometheus_exporter

✨ Features: Native OpenMetrics · Multiple endpoints · Basic Auth & TLS · Global labels · YAML config · Clean Architecture

📋 Table of Contents

• ✨ Features
• 🚀 Get started
• ⚙️ Configuration & development
• 📊 Dashboards & alerts
• 📸 Screenshots
• 🔐 Security & supply chain
• 🤖 Automation
• 🤝 Contributing
• 📜 License

✨ Features

• ✅ Wide metric coverage: nodes, partitions, jobs, CPUs, GPUs, scheduler internals (sdiag RPC stats), fairshare, reservations, licenses, per-user/per-account roll-ups.
• ✅ All 16 collectors are optional and toggle via --collector.<name> / --no-collector.<name> flags.
• ✅ GPU metrics per account and user (slurm_account_gpus_running, slurm_user_gpus_running) — covers --gres, --gpus, and --gpus-per-node jobs.
• ✅ Per-reservation node state metrics (slurm_reservation_nodes_*).
• ✅ TLS + Basic Authentication via --web.config.file.
• ✅ OpenMetrics format (exemplars, Prometheus 2.x+ features).
• ✅ Per-collector health metrics (slurm_exporter_collector_success, slurm_exporter_collector_duration_seconds).
• ✅ Liveness probe at /healthz for Kubernetes / systemd orchestration.
• ✅ Ten ready-to-use Grafana dashboards + site-neutral Prometheus alerting rules.
• ✅ Multi-arch Docker images (linux/amd64 + linux/arm64), signed with cosign keyless, CycloneDX SBOM per release.
• ✅ Goreportcard A+ (100% across gofmt, go vet, gocyclo, ineffassign, misspell, license).

---

🚀 Get started

The fastest path is the published Docker image — assuming the host has a working slurm-client + munged setup (slurmctld host, login node, or a monitoring VM already enrolled in the cluster):

docker run -d --name slurm_exporter \
  -p 9341:9341 \
  -v /etc/slurm:/etc/slurm:ro \
  -v /var/run/munge:/var/run/munge:ro \
  -v /etc/munge/munge.key:/etc/munge/munge.key:ro \
  sckyzo/slurm-exporter:latest

curl http://localhost:9341/metrics | head

Then point Prometheus at :9341/metrics (sample scrape config in monitoring/).

For everything else — compose / Kubernetes / a remote monitoring node, or running the binary directly on a node — pick one of the three paths below.

🐳 Docker images

Two variants, both published as multi-arch manifests (linux/amd64 + linux/arm64) to Docker Hub (sckyzo/slurm-exporter) and GHCR (ghcr.io/sckyzo/slurm_exporter):

Pre-release tags (vX.Y.Z-rc1 etc.) push only the pinned version and never overwrite the floating aliases.

Full Docker reference — compose, Kubernetes patterns, env-var overrides, version compatibility, troubleshooting — in docker/README.md.

📥 Pre-compiled binary

Linux, macOS, and Windows binaries (amd64 / 386 / arm64) on the Releases page. Each archive ships with a CycloneDX SBOM and a cosign-verifiable checksum file.

Installing as a systemd service:

# 1. Grab and install the binary
tar -xzf slurm_exporter-*-linux-amd64.tar.gz
sudo mv slurm_exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/slurm_exporter

# 2. Install the unit file (adapt User / ExecStart for your environment)
sudo cp systemd/slurm_exporter.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now slurm_exporter

🔨 From source

git clone https://github.com/sckyzo/slurm_exporter.git
cd slurm_exporter
make build

The binary lands in bin/slurm_exporter. See CONTRIBUTING.md for the full development setup (Go 1.26+, golangci-lint, the containerized make check / make report targets).

---

⚙️ Configuration & development

Make targets

make check, make report, and make report-deps run inside a container — contributors only need Docker, no host Go install required.

---

📊 Dashboards & alerts

All monitoring assets live under monitoring/:

monitoring/
├── grafana/dashboards/    10 Grafana dashboards (JSON) + screenshots
└── prometheus/
    ├── alerts.yml         Alerting rules (severity-based, site-neutral)
    └── rules.yml          Recording rules

End-to-end wiring (Prometheus scrape config, rule_files, Alertmanager) in monitoring/README.md.

Grafana dashboards

Ten dashboards, Grafana 12+, all using a $datasource template variable for portability.

Import via Grafana UI, provisioning, or API — see monitoring/grafana/dashboards/README.md for the three options.

Scale note: On 100k+ node clusters, always pick a specific partition on the Node Detail dashboard via the $partition variable. The partition summary and the Down/Drain panels are always O(partitions).

Prometheus alerts & recording rules

Starter set in monitoring/prometheus/alerts.yml (severity-based, site-neutral): node down/drain/maint, partition nodes down, pending-job queue backlog (warn/crit), job failure rate (warn/crit), slurmctld cycle slowness, SlurmDBD queue backlog, GPU saturation. One supporting recording rule in monitoring/prometheus/rules.yml.

Threshold table, calibration guidance, and validation recipes in monitoring/prometheus/README.md.

# Validate before deploying
promtool check rules monitoring/prometheus/alerts.yml monitoring/prometheus/rules.yml

Site-specific labels (team, runbook_url, dashboard_url) are intentionally omitted — add them via Prometheus external_labels or Alertmanager routing.

---

📸 Screenshots

Screenshots taken on a 20-node test cluster (alice/bob/carol/dave/eve/frank, multiple accounts and partitions). Click any thumbnail to open the full-size image. See monitoring/grafana/dashboards/README.md for the full dashboard documentation.

---

🔐 Security & supply chain

Found a vulnerability? See SECURITY.md for how to report it privately.

Every published artifact carries verifiable provenance and is scanned for known vulnerabilities before release.

• 🖋️ Signed container images — every Docker manifest is signed via cosign keyless (Sigstore / Fulcio). The signing identity is the GitHub Actions workflow itself, attested by the runner's OIDC token. Verify with:

  cosign verify sckyzo/slurm-exporter:latest \
    --certificate-identity-regexp 'https://github.com/SckyzO/slurm_exporter/.github/workflows/release.yml@.*' \
    --certificate-oidc-issuer https://token.actions.githubusercontent.com
  

• 🧾 Signed release checksums — slurm_exporter_checksums.txt ships with a .sigstore.json Sigstore bundle (certificate + signature) for offline verification of every release archive. Verify with cosign verify-blob --bundle slurm_exporter_checksums.txt.sigstore.json slurm_exporter_checksums.txt.
• 📦 CycloneDX SBOMs — one *.sbom.json per release archive lists every Go module compiled in (with versions and PURLs). Suitable for Dependency-Track, Anchore Enterprise, and similar.
• 🛡️ Vulnerability scanning — Trivy scans both Docker variants on every PR that touches Dockerfile*, go.mod, or go.sum. PRs are blocked on HIGH/CRITICAL CVEs that have an upstream fix. A weekly cron re-scans the published images so post-release CVEs surface as workflow failures.
• 👤 Non-root by default — the standard image runs as slurmexporter (uid 9341, gid munge); the minimal image runs as nonroot (uid 65532). Example compose drops all capabilities, mounts read-only, no-new-privileges.
• 🪞 Distroless variant — the :latest-minimal tag runs on gcr.io/distroless/cc-debian12:nonroot: no shell, no package manager, no userland beyond the dynamic loader and libstdc++. Smallest viable attack surface for a binary that has to dlopen libmunge at runtime.
• 🔁 Reproducible build chain — binaries built with pinned Go 1.26.8 in CI; Docker images from pinned ubuntu:26.04 / gcr.io/distroless/cc-debian12:nonroot / debian:13-slim (libmunge extractor). All version bumps go through Dependabot PRs.

Detailed verification recipes (cosign for blobs, SBOM inspection, image labels) in docker/README.md.

---

🤖 Automation

The repo runs a few autonomous workflows so dependencies and images stay fresh without manual babysitting:

• Dependabot weekly — Monday 05:00 Europe/Paris, four ecosystems (Go modules, GitHub Actions, two Docker base images). Related deps grouped (golang.org/x/*, github.com/prometheus/*, etc.).
• make report-deps — on-demand tabular snapshot of every Go module (direct + indirect) with patch/minor/major bump classification. Runs in the containerized toolchain, no host Go required.
• Trivy weekly scan — Monday 06:00 UTC against the published images; CVE regressions show as workflow failures.
• Docker Hub README sync — docker/README.md is mirrored to the Docker Hub repo description on every push to master (and on every release).
• Auto Docker image refresh — every release tag triggers GoReleaser, builds both variants for both architectures, pushes to GHCR + Docker Hub, signs every manifest, and emits SBOMs.

---

🤝 Contributing

PRs and issues welcome. Before sending a contribution:

• Read CONTRIBUTING.md — covers the Definition of Done, code conventions (initialisms, collector pattern, test fixtures), and the Common Pitfalls section (truncation gotcha on squeue -O field: / sinfo --Format, multi-arch path differences, etc.).
• Run make check (containerized vet + lint + test) and make report (offline goreportcard, must stay ≥ B) before opening a PR.
• One issue → one branch → one PR. Don't mix refactoring and new features in the same change.

The release process and the validation playbook live in docs/release-process.md and docs/validation-checklist.md.

---

📜 License

This project is licensed under the GNU General Public License, version 3 or later.

---

🍴 About this fork

Fork of cea-hpc/slurm_exporter, itself a fork of vpenso/prometheus-slurm-exporter (now apparently unmaintained).

Related project (not a replacement): sckyzo/slurm_prometheus_exporter is a separate, from-scratch exporter built around Slurm 25.11+'s native OpenMetrics endpoint — a different tool with a different scope. This exporter stays actively maintained for the broad metric and dashboard coverage the native endpoint doesn't provide.