Highway

Build distributed, event-driven microservices on .NET.

---

Highway gives you durable queues, publish/subscribe and RPC, Recurring jobs — all over one broker you run yourself, in one process.

No AWS concepts. No Azure. No RabbitMQ, no Service Bus, no gRPC, no Kafka, no Redis, No Hangfire, no Quartz, no cron container. No connection strings to a managed service, no SDK-shaped abstractions leaking into your domain. You write plain C# POCOs — a class per message — and Highway builds the system around them: service discovery, load balancing, durable delivery, retries, timeouts and serialization.

The broker Highway Server is a high-performance, low-footprint, distributed .NET 10 server (and an embedded in-process server for development and tests), so every call is a round trip through a local store rather than a hop into somebody else's cloud. It needs the .NET 10 SDK and nothing else — no Docker, no external infrastructure, not even for the integration tests.

2.0. The broker now runs on a purpose-built stack — a solid embedded storage engine behind a Highway-native RESP server (Garnet is gone) — with replication and client-herd failover, and a built-in IDistributedCache (with HybridCache L2) so caching needs no separate Redis. The packages ship on nuget.org as 2.0.0. See Status and Known limits.

---

Durable queues

Work that must happen exactly once, survive a restart, and wait for a consumer that may not be running yet.

[Queue("email.send")]
public sealed class SendEmail : ISend
{
    public string To   { get; set; } = "";
    public string Body { get; set; } = "";
}

public sealed class EmailProcessor : IProcess<SendEmail>
{
    public Task ProcessAsync(SendEmail message, CancellationToken ct = default)
        => _smtp.SendAsync(message.To, message.Body, ct);
}

await client.SendAsync(new SendEmail { To = "user@example.com", Body = "..." });

Sending never requires a running processor. The message waits in the queue until one claims it — through a deployment, a crash, or a mailer that is simply down. Delivery is at-least-once; mark a message [Idempotent] to have Highway suppress redelivery.

Three instances of a processor share the work. Failures retry with backoff and land in a dead-letter queue that tells you why.

---

Publish / subscribe

Facts that several parts of the system care about, each independently.

[Channel("users.signedup")]
public sealed class UserSignedUp : IPublish
{
    public int UserId { get; set; }
}

public sealed class SendWelcomeEmail : ISubscribe<UserSignedUp>
{
    public Task SubscribeAsync(UserSignedUp message, CancellationToken ct = default) => ...;
}

await client.PublishAsync(new UserSignedUp { UserId = 42 });

Every subscription group gets its own copy. Three instances of a subscriber each receive the message; three instances of a processor split the work between them. That difference is the reason both verbs exist.

A subscriber that is down receives what it missed when it returns.

---

RPC

When you need the answer before you can continue.

[Service("orders.create")]
public sealed class CreateOrder : IReturn<OrderResult>
{
    public int CustomerId { get; set; }
    public string Item    { get; set; } = "";
}

public sealed class OrderResult : Output      // Output carries StatusCode and Error
{
    public string? OrderId { get; set; }
}

public sealed class CreateOrderService : AsyncService<CreateOrder, OrderResult>
{
    public override Task<OrderResult> ExecuteAsync(CreateOrder request, CancellationToken ct = default)
        => Task.FromResult(new OrderResult { OrderId = "ORD-1", StatusCode = 200 });
}

var result = await client.ExecuteAsync(new CreateOrder { CustomerId = 7, Item = "WIDGET" });
if (result.StatusCode != 200) { /* handle it — errors are data, not exceptions */ }

Highway routes the call to whichever node hosts the service and balances across them. You get an answer or a timeout, never silence. ExecuteAsync does not throw on a service failure — errors arrive as StatusCode and Error, so a failing dependency is a branch in your code rather than a catch block.

Choosing between the three is one sentence: one handler → SendAsync, many handlers → PublishAsync, need the answer → ExecuteAsync.

---

High availability via replication

Run more than one broker and Highway tolerates a server failure without losing acked work. Standbys follow the primary by WAL shipping; on a failure the model is client-herd mastership — the master is simply the node the client herd is connected to, and the herd converges on the highest-priority reachable successor.

• No elections, no quorum, no votes. Succession is a deterministic priority order, and epoch fencing guarantees at most one writable master — no split-brain.
• Clients replay their own in-flight work with the same request id across the failover, so a call in flight when the master dies gets its answer from the successor, not a duplicate or a loss.
• Graceful handover (HW.REPL.GOODBYE) drains a node before it steps down; failover is automatic and there is no auto-failback to surprise you.

Turn it on with HighwayReplicationOptions (start-as-replica, a primary to follow, a priority). The RPO is the async-replication lag window — bounded and reported, never silent (C9); see Replication & Failover.

---

Also included

• Recurring jobs — a schedule that sends a queue message. o.Jobs.Daily<GenerateStatements>(new TimeOnly(2, 0)), o.Jobs.Every<ReconcileLedger>(TimeSpan.FromMinutes(15)), or a five-field cron expression. Schedules survive restarts; missed occurrences collapse to one catch-up fire.
• Distributed cache — opt in with AddHighwayCache() and Highway provides an IDistributedCache (and a HybridCache L2) backed by a broker-local, never-replicated store, so caching needs no separate Redis. It is cold after a failover and TTL-bounded — a cache, not a second source of truth.
• A dashboard — embedded in the broker, on its own port. Live message flow, a service catalogue, queue depths, dead letters and a flight recorder you can replay.
• Delayed delivery, dead letters, idempotency, graceful node decommissioning — the reliability machinery, in the box.

---

Installation & Quick Start

Highway is distributed via NuGet for application libraries and GitHub Releases for standalone broker binaries:

1. Add NuGet Packages

In your domain contracts library (zero dependencies):

dotnet add package Highway.Abstractions --prerelease

In your application / worker service:

dotnet add package Highway.Client --prerelease

2. Register Highway in DI

builder.Services.AddHighway(o =>
{
    o.NodeName = "my-service-1";
    o.Server   = "127.0.0.1:6500";
});

Assembly scanning finds all ISend, IPublish, AsyncService, and ISubscribe implementations automatically.

---

Contributor & Sample Workflow

To run the sample applications or contribute to Highway:

git clone https://github.com/System-D-AB/highway.git
cd highway
dotnet build Highway.slnx

Run the three sample processes, one per terminal:

dotnet run --project samples/Highway.Samples.Broker        # the broker  :6500
dotnet run --project samples/Highway.Samples.OrderService  # hosts services
dotnet run --project samples/Highway.Samples.Storefront    # calls them

In the storefront try invoice ORD-1 (queue), low WIDGET 2 (pub/sub) and order 2 WIDGET (RPC). Start a second order service and watch queue work get shared while published events reach both. See samples/README.md.

---

Documentation

---

Status

2.0 — released. Queues, pub/sub, RPC, dead letters, delayed delivery, recurring jobs, dashboard, authentication, TLS, and NuGet packaging (2.0.0) all ship today, now on a purpose-built storage-engine + RESP broker (Garnet removed in feature 041). 2.0 adds replication with client-herd failover (features 042 + 042-1 — WAL-shipping standbys, epoch fencing, no elections; the master is the node the client herd is on) and an opt-in broker-local cache (feature 044 — IDistributedCache/HybridCache L2, never replicated). More than 1,200 tests pass.

Not yet done: metrics (Meter) and health endpoints, and prebuilt broker distributions for Linux and macOS. The Windows (win-x64) distribution — the highways zip with service installers — ships as of 2.0.

---

Known limits

Highway declines to promise these, and says so rather than letting you find out:

• No exactly-once delivery. At-least-once, with [Idempotent] to suppress redelivery.
• The cache is broker-local and never replicated. It is cold after a failover and epoch-invalidated — a cache miss is one more trip to the system of record, not data loss (C10).
• Not a replayable log. Pub/sub does not retain history for groups that never registered.
• No transactional enlistment, message priority, or per-message TTL.
• No characterised throughput. No benchmark exists, so no figure is claimed anywhere.
• The prebuilt broker distribution is Windows (win-x64) only. Linux and macOS are fully supported from source (a self-contained dotnet publish), but no prebuilt zip ships for them yet.
• Retention over time is still unbounded (C4.1, awaiting a breaking framing change). Storage growth itself is now bounded — the storage engine's compaction reclaims consumed messages (C4.6, met on the current engine).

Every one is a numbered row in constraints.md with an implementation status, so intent and reality can be compared line by line.

---

Building and testing

dotnet build Highway.slnx --no-incremental   # expected: zero warnings
dotnet test Highway.slnx                     # ~7 minutes

Integration tests run against a real embedded broker in-process — no Docker, no external infrastructure.

---

License

MIT.

Highway's broker uses RocksDB for storage (via the RocksDB .NET binding) and StackExchange.Redis (MIT) as the RESP client. One RESP output-formatter source file is vendored from Microsoft Garnet (MIT) with its copyright header intact — see THIRD-PARTY-NOTICES.md. Garnet itself is no longer a dependency (removed in feature 041).