NVIDIA Infra Controller (NICo)

Repository move notice: On September 4, 2026, the NICo repository moved from the NVIDIA GitHub organization to dsx-ai-factory/infra-controller. Existing repository URLs and standard Git operations continue to work through GitHub redirects. If you maintain automation or integrations that reference NVIDIA/infra-controller, such as GitHub Actions, webhooks, or pinned repository URLs, update them to dsx-ai-factory/infra-controller.

What Is NICo

NICo delivers lifecycle automation for bare-metal systems that secures datacenter infrastructure at its foundation. It is an open source suite of microservices that run next to the hardware, managing and automating the full bare-metal lifecycle: hardware discovery, firmware validation, BlueField DPU provisioning, network isolation, tenant provisioning, and secure sanitization between tenants. NVIDIA Cloud Partners (NCPs) and infrastructure operators can use NICo to stand up and operate AI factory-scale infrastructure through APIs instead of runbooks.

Full component diagram: every Site Controller service, host agent, and off-the-shelf dependency

AI factories need rack-level management, host lifecycle automation, and workload isolation that general-purpose tools do not provide as one integrated system. Without it, operators are left with manual discovery and firmware alignment on every rack bring-up, no single enforcement point for tenant isolation across Ethernet, InfiniBand, and NVLink, custom scripts for sanitization and attestation, and firmware drift across hardware generations.

NICo closes these gaps. In the default configuration, a managed host is a server with one or more BlueField DPUs, and the DPU is the enforcement boundary. NICo provisions and controls the DPU directly, independently of whatever the tenant runs on the host, so the layers above it can treat bare metal as a reliable building block. NICo also supports zero-DPU hosts, where the DPU policy is ignore or nic. Their tenant instances attach to Flat VPCs on the underlay, and the operator's fabric provides the isolation. Refer to the glossary for both host types.

What NICo Does

For GB200 NVL72 systems, NICo Flow extends all of this to the rack. It treats racks and NVL domains as first-class entities and sequences power and firmware operations across trays. NICo also gates instance allocation on NVLink fabric health. Refer to Key Capabilities for the complete list and to the Hardware Compatibility List for supported servers and DPUs.

Who Uses NICo

NICo REST authorizes every caller as a provider or a tenant, or as a service account that acts as both. All of them work through the same REST API, and every endpoint is documented in the REST API Reference, including the typical API call flows for each persona.

Service Account mode is the third option. NICo REST can be configured so that one organization holds both the provider and tenant roles through a service account. This suits deployments that manage tenancy in a system above NICo and call it as one principal. Refer to Tenant Management with Keycloak for the service-account client setup and to the authorization roles in the glossary for what each role family can do.

The Lifecycle NICo Automates

NICo organizes bare-metal management into three phases. Day 0 takes hardware from "racked and cabled" to "ready for tenants". Day 1 isolates and provisions a host for a tenant. Day 2 keeps the fleet healthy and returns released hosts to the pool.

Day 0: Discovery, Validation, and Ingestion: The provider registers each host as an Expected Machine with its BMC MAC address, chassis serial, and factory BMC credentials. From there NICo takes over: it discovers the host over Redfish, links each DPU to its host, validates the machine against its SKU, runs burn-in and fabric tests, brings firmware to the site baseline, installs the DPU OS and HBN, attests the host with measured boot and TPM checks when attestation is enabled, and allocates its IP addresses. Component inventory is collected automatically, so no manual component entry is required.

Day 1: Isolation, Lockdown, and Provisioning: Before a tenant receives a host, NICo establishes isolation on every network plane, locks down UEFI and the BMC, and coordinates the iPXE boot of the tenant's chosen OS image. The provider grants the tenant an allocation of hosts, and the tenant creates instances from it through the REST API.

Day 2: Operations, Health, and Tenant Transitions: NICo monitors health continuously, schedules firmware upgrades on unoccupied hosts, upgrades itself in place, and sanitizes every released host before it returns to the available pool, re-attesting it when attestation is enabled.

Refer to Day 0, Day 1, and Day 2 Lifecycle for the detail behind each step.

Typical Flows

Each flow below is an end-to-end procedure you can run against a site. The provider-side flows use nicocli or the REST API, and the REST API Reference documents the matching API call sequences.

Where NICo Fits

NICo sits below Kubernetes and platform layers. It exposes REST and gRPC APIs that higher-level systems consume directly, and it does not dictate how scheduling, tenancy policy, or workloads are managed above it.

How NICo Relates to Other Tools

• Kubernetes, SLURM, and workload schedulers run above NICo. NICo provisions hosts and hands them off at boot. Assembling them into clusters, scheduling workloads, and patching the OS belong to the platform layer.
• BMaaS, VMaaS, and ISV control planes consume the NICo REST and gRPC APIs to allocate and provision bare metal, and they own tenancy policy.
• UFM, NMX-C, and the switch fabric stay under the network team. NICo calls the UFM and NMX-C APIs for InfiniBand and NVLink partitions and programs isolation on the DPU, but it does not configure switches or install UFM, NMX-C, or Cumulus Linux.
• DPF and RMS are NVIDIA components that NICo integrates with. The DOCA Platform Framework provisions DPUs for NICo, and the Rack Management Service carries out rack-level hardware operations that Flow sequences.
• PostgreSQL, Vault, Temporal, cert-manager, and the telemetry stack are off-the-shelf dependencies. NICo uses them but does not operate them.

Refer to Scope and Boundaries for the full division of responsibility.

How It Is Built

The Site Controller is the on-site control plane: Rust services deployed on a Kubernetes cluster of at least three nodes, all talking to each other over mutual TLS (mTLS) and gRPC, plus the Go Site Agent that links the site to NICo REST. NICo REST is a Go service layer that exposes those capabilities as a REST API. It can run next to the Site Controller or centrally in the cloud, where many sites connect to it through their Site Agents.

Refer to Architecture: Overview and Components for the full component list and to the state handling design for how the state machines work.

Installation

There are two ways to run NICo. Use the local development stack to work on the code against mock hardware. Use the production installer to deploy a complete site onto a bare-metal Kubernetes cluster.

Local Development

The fastest way to run the whole stack locally is DevSpace on a kind cluster. It compiles the Core and REST images from this clone, deploys the Helm charts, and uses Machine-a-Tron to supply mock hosts, so you can exercise the REST API and the Core gRPC API end to end without hardware.

# 1. Install Docker, kind, kubectl, helm, and devspace. DevSpace builds every
#    image inside Docker, so no Rust or Go toolchain is needed for this path.

# 2. Create a local cluster. The bootstrap script works on your current
#    Kubernetes context, so this must exist first.
kind create cluster --name nico

# 3. Bootstrap the cluster-side prerequisites (cert-manager, PostgreSQL, Vault,
#    Temporal, and Keycloak) on that context. Safe to re-run.
dev/deployment/devspace/bootstrap-prereqs.sh

# 4. Build and deploy NICo Core, NICo REST, and the mock hosts.
devspace deploy

For a plain build and test loop without a cluster, install rustup with the Rust version pinned in rust-toolchain.toml, the Go version in rest-api/go.mod, and cargo install cargo-make, then:

cargo build                              # all Rust workspace crates
cargo make correctly-execute-tests       # build prerequisites, then run the Rust tests
make rest-build                          # Go REST API binaries
make rest-test                           # Go unit tests with PostgreSQL and mock gRPC servers
cargo make pre-commit-verify-workspace   # what CI runs before merge

On a fresh Ubuntu VM, dev/deployment/devspace/setup-devspace-on-host.sh performs all four steps, including the kind cluster.

Refer to Local Development with DevSpace for profiles, port-forwarding, and teardown, to the Development page for host package prerequisites, to the Build Guide for ARM64 and release builds, and to AGENTS.md for the full build, test, and lint command reference.

Production Deployment

helm-prereqs/setup.sh deploys the full NICo stack onto a bare-metal Kubernetes cluster in layers. The first three always install. DPF and RMS install by default and can be skipped. Flow installs whenever REST does.

Refer to What gets deployed for the complete list with chart versions.

Prerequisites

• A running Kubernetes cluster of at least three nodes with KUBECONFIG set
• helm, helmfile, kubectl, and jq installed
• Images pushed to your container registry

Quick Start

# 1. Build and push amd64/arm64 container images from this clone.
#    Refer to docs/manuals/building_nico_containers.md for the build-host prerequisites.
#    The values below are examples: replace every one of them with your own.
export IMAGE_REGISTRY=my-registry.example.com/infra-controller
export IMAGE_TAG=v2.2.0
make images IMAGE_REGISTRY="${IMAGE_REGISTRY}" IMAGE_TAG="${IMAGE_TAG}"   # NICo Core and REST service images

# 2. Set environment variables, reusing the registry and tag from step 1.
export KUBECONFIG=/path/to/kubeconfig
export NICO_IMAGE_REGISTRY="${IMAGE_REGISTRY}"
export NICO_CORE_IMAGE_TAG="${IMAGE_TAG}"
export NICO_REST_IMAGE_TAG="${IMAGE_TAG}"
# For an authenticated registry, read the pull secret from a prompt so it
# never appears in shell history:
# read -r -s -p 'Registry pull secret: ' REGISTRY_PULL_SECRET && export REGISTRY_PULL_SECRET

# DPF (DOCA Platform Framework) DPU provisioning installs by default.
# Set these two variables, or pass --skip-dpf to opt out:
export NICO_DPF_DPU_INTERFACE=ens1f0          # controller NIC facing the DPUs
export NICO_DPF_DPU_CLUSTER_VIP=192.0.2.10    # free routable IP for the DPU cluster control plane

# RMS (Rack Management Service) installs by default and requires an image
# tag. Its default image is entitlement-gated on NGC, so also supply an NGC
# API key (it defaults to REGISTRY_PULL_SECRET), point NICO_RMS_IMAGE_REPO
# at your mirror, or pass --skip-rms to opt out:
export NICO_RMS_IMAGE_TAG=v0.8.0
read -r -s -p 'NGC API key for RMS: ' NICO_RMS_NGC_API_KEY && export NICO_RMS_NGC_API_KEY

# 3. Customize site-specific values
#    Edit helm-prereqs/values/nico-core.yaml:
#      nico-api.hostname      - your site's external API hostname
#      nico-api.siteConfig    - network pools, VLAN ranges, IB config, MetalLB VIPs
#    Edit helm-prereqs/values/metallb-config.yaml:
#      IPAddressPool, BGPPeer - your site's VIP ranges and TOR switch config
#    Edit helm-prereqs/values.yaml:
#      siteName               - short site identifier

# 4. Run setup. It installs common services, NICo Core, and NICo REST in order.
cd helm-prereqs
./setup.sh                # interactive: prompts before deploying Core and REST
./setup.sh -y             # non-interactive: deploys everything including DPF (CI/CD)
./setup.sh -y --skip-dpf  # non-interactive: skip DPF (no DPUs, or still on iPXE)

To tear everything down:

cd helm-prereqs
./clean.sh

Refer to the Quick Start Guide for the step-by-step walkthrough, including the siteConfig block and first host discovery, and to helm-prereqs/README.md for the full reference: PKI architecture, PostgreSQL setup, phase-by-phase description, every environment variable, DPF options, secrets, health checks, and troubleshooting.

Examples

Runnable inputs and reference configurations that ship with the repository:

Repository Tour

Roadmap

NICo publishes a three-month rolling roadmap as a pinned GitHub issue, maintained alongside the monthly release cadence. It is directional rather than a commitment, and each item links to its tracking issue.

• Public roadmap
• Milestones for the current implementation status of each release
• Release Policy for how the roadmap and the release cycle fit together

Community and Support

• Chat: join the NICo community Slack
• Questions and ideas: GitHub Discussions
• Bugs and feature requests: GitHub Issues
• Releases: Release Notes for what changed, and the Release Policy for the monthly branch, version, and support cadence
• Security: report vulnerabilities as described in SECURITY.md

Contributing

Contributions are welcome. Refer to the contribution guide for the development environment, the DCO sign-off process, and the engineering guidelines, and to the code of conduct for contributor expectations.

License and Release Notice

NICo is licensed under the Apache License 2.0.

NICo is under active development. The software is provided "as is" without warranties of any kind. Features, APIs, and configurations may change in future releases. For production deployments, please test thoroughly in non-critical environments first.