Reflex

Reflex is a Rust library for control loops that act on observability data. Metrics queried from Datadog and forecasts from Toto become typed state. A model, such as TypeSafe AI's Jev, recommends an action, and Reflex commits it only if it passes the guards and invariants you declared.

Application supplies typed state
              ↓
Judge recommends an action
              ↓
Executor checks legality, guards, and invariants
              ↓
Verified state transition
              ↓
Optional effects perform external work

When to use it

Reflex fits control loops where the right action depends on changing conditions, but execution must obey fixed constraints.

• Circuit breaking. Open, probe, or close based on service health, while enforcing cooldowns and probe limits.
• Scheduling. Choose which request gets capacity next, while enforcing resource limits.
• Autoscaling. Add or remove capacity based on demand and forecasts, while enforcing capacity bounds.
• Retries. Decide whether to retry a failed request and how long to back off, while enforcing retry budgets and a maximum backoff.

A circuit breaker

The repository includes a simulator that runs Reflex against simulated services. The sections below walk through its circuit breaker, starting with the state machine and then the running simulation.

A gateway routes client traffic to three services, Catalog, Payments, and Search, each behind its own circuit. Jev decides when a circuit should open or probe. Reflex decides whether that decision may take effect.

Each recommendation is a typed action. Besides Jev's choice, it carries the Datadog evidence the decision was based on, when that evidence was observed, and the circuit revision it applies to.

struct Action {
    choice: Choice,                          // Open, Probe, or NoChange
    telemetry: Option<TelemetryEvidence>,    // Datadog observations Jev received
    observed_at: f64,
    revision: u64,
}

These are the machine's main rules, abridged from jev.rs.

let definition = state_machine! {
    phase: CircuitPhase,
    data: Data,
    action: Action,
    event: Event,
    invariants: [valid_phase],
    transitions: [
        // Jev's recommendations
        CircuitPhase::Closed + action(Action { choice: Choice::Open, .. }) => CircuitPhase::Open {
            guard: can_open, update: open,
        },
        CircuitPhase::Open + action(Action { choice: Choice::Probe, .. }) => CircuitPhase::HalfOpen {
            guard: can_probe, update: probe,
        },
        CircuitPhase::Closed + action(Action { choice: Choice::NoChange, .. }) => CircuitPhase::Closed {
            guard: fresh,
        },
        // Probe results
        CircuitPhase::HalfOpen + event(Event::Response(Observation { outcome: ClientOutcome::Success, .. })) => CircuitPhase::HalfOpen {
            guard: current_probe, update: probe_succeeded,
        },
        CircuitPhase::HalfOpen + event(Event::FinalProbe(Observation { outcome: ClientOutcome::Success, .. })) => CircuitPhase::Closed {
            guard: final_probe, update: close,
        },
        CircuitPhase::HalfOpen + event(Event::Response(Observation { outcome: ClientOutcome::Error | ClientOutcome::Timeout, .. })) => CircuitPhase::Open {
            guard: current_probe, update: reopen,
        },
        // Evaluation failed (for example, an inference error): leave the circuit as it is
        CircuitPhase::Closed + evaluation_error(_) => unchanged {},
        // ... clock ticks, response recording, and the remaining no-change rules
    ],
};

The guards are where observability data meets fixed rules.

The valid_phase invariant checks that the phase, cooldown, and probe reservation always agree. Guards run at execution time against current state, so a recommendation based on delayed telemetry or an inaccurate forecast cannot open, probe, or close a circuit on its own.

Running the simulation

During a traffic surge, Jev opens the Payments circuit. After the cooldown, it probes, and the circuit closes once traffic recovers and five probes succeed. The panel on the right compares a Toto forecast of Payments request rate with the Datadog observations that followed.

Where the state comes from. The simulated services publish metrics to Datadog. The simulator queries them back to build each service's state, such as its error rate, latency, and queue depth.

How it is forecast. A local Toto service receives the observed history of request rate, queue depth, and utilization, and forecasts the next 120 seconds as p10/p50/p90 values in ten-second buckets. It needs 320 seconds of history before the first forecast. Select a service to compare the frozen forecast with what was later observed.

What happens to a decision. Every 10 seconds, Jev evaluates each service and returns Open, Probe, or NoChange. Reflex applies the recommendation only if the matching transition's guard passes. Open Activity and inspect a decision to see the Datadog values and timestamps Jev received, any forecast, and whether the transition was applied or rejected, with the rejection reason.

The scenario shown. In Cyclical load · Toto, Payments repeats a two-minute pattern for ten minutes. Traffic rises 4× at +30s, service time rises 6× at +60s, and both recover at +90s. Jev chooses when to open and probe. Reflex enforces the cooldown and the five-probe close.

Reflex also emits its own OpenTelemetry counters and spans (reflex.evaluations, reflex.transitions). The simulator exports these to Datadog with the application metrics, and the supplied dashboards show both. The simulator also includes a resource scheduler that follows the same pattern; see the simulator guide.

Run the simulator

Build the UI once.

npm ci --prefix crates/reflex-sim/ui
npm run build --prefix crates/reflex-sim/ui

In one terminal, start the Toto service.

uv sync --project integrations/toto --python 3.12 --locked
uv run --project integrations/toto --locked reflex-toto

Wait for Ready: http://127.0.0.1:8765. The first start downloads a pinned Toto-2.0-22m checkpoint, which runs on CPU by default.

In a second terminal, copy .env.example to .env.local, fill in your keys, and start the simulator.

set -a
. ./.env.local
set +a
cargo run -p reflex-sim --features datadog --locked -- \
  --playground --policy jev --datadog --datadog-evidence \
  --toto-url http://127.0.0.1:8765

Select Cyclical load · Toto → Run, then select Payments. Use the decision's simulation_run to filter the Datadog dashboards. Press Ctrl+C to stop and flush telemetry.

There are other ways to run it.

• To run without Datadog, drop --features datadog, --datadog, and --datadog-evidence. State and forecasts then come from the simulator's local observations, and only TYPESAFE_API_KEY is required.
• To publish to Datadog but decide on local state, use --datadog without --datadog-evidence. Publishing needs only DD_API_KEY.
• To run without credentials, use cargo run -p reflex --example circuit_breaker, a smaller circuit breaker with a deterministic judge.

Use Reflex in your application

Your application gathers the evidence, whether that is a Datadog query, a forecast, or local measurements, and builds the typed state passed to the judge.

Your application calls the controller, then the executor.

// Gather evidence (for example, Datadog observations and a Toto forecast) into `state`.
let evaluation = controller.evaluate(&state).await;
let outcome = executor.execute(evaluation).await?;

Pass the entire evaluation result to the executor; the machine can define transitions for inference failures as well as recommendations. State transitions commit in memory, and external effects run afterward. See execution semantics for persistence and failure handling.

For a smaller, self-contained machine that runs without credentials, see the circuit-breaker example and the SDK guide.

The crates are not yet published to crates.io. Add path dependencies on your checkout.

[dependencies]
reflex = { path = "../reflex/crates/reflex" }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

To use Jev, also add the typesafe-ai and reflex-typesafe crates and inject a TypeSafeClient into TypeSafeJudge. The live Jev example shows the setup (cargo run -p reflex-typesafe --example jev, with TYPESAFE_API_KEY set). See the SDK guide for hook signatures, executor construction, and the TypeSafe integration.

Further reading

Datadog and Toto

• Datadog control loop, covering metrics, warmup, delays, and troubleshooting
• Forecasting, covering observation series, timing, uncertainty, and example workloads
• Local Toto service, covering the API contract, model selection, and offline use
• Datadog dashboards
• Simulator scenarios and simulation model

Reflex SDK

• SDK quickstart and API behavior
• Core concepts and circuit-breaker walkthrough
• Declarative state-machine design
• Core metrics, traces, and logs
• Standalone TypeSafe client instrumentation
• Capacity research on workloads, algorithms, and methods

Run the workspace tests with these commands.

npm ci --prefix crates/reflex-sim/ui
npm run build --prefix crates/reflex-sim/ui
cargo test --workspace --all-targets --locked
cargo test --workspace --doc --locked

License

Reflex is licensed under Apache-2.0. See NOTICE for attribution. Third-party dependencies and assets retain their own licenses.

Third-party components are listed in LICENSE-3rdparty.csv. See the inventory notes for coverage, sources, and unresolved entries.

Ownership

Owned by Datadog, Inc. See maintenance responsibilities.