Remoc 🦑 — Rust RPC with remote channels and objects

Remoc is an async RPC and IPC library for Rust. Unlike conventional RPC, calls can pass live Tokio-style channels and remote objects that remain usable after the call returns.

Over a single underlying transport, such as TCP or TLS, it provides:

• calling of trait methods on a remote object (RPC) and of remote functions,
• multiple channels of different types like MPSC, oneshot, watch, etc.,
• remote synchronization primitives,
• remotely observable collections.

Remoc is written in 100% safe Rust, builds upon Tokio, and uses Serde with the compact, forward- and backward-compatible Postbag binary codec. Remoc does not depend on any particular transport type, so it serves for inter-process communication over a UNIX socket or pipe as well as over the network.

See remoc.rs for an illustrated overview, benchmarks, examples, and an RPC library comparison.

Getting started · Remote procedure calls · Channels · When to use Remoc

Introduction

A common pattern in Rust programs is to use channels to communicate between threads and async tasks. Setting up a channel is done in a single line and it largely avoids the need for shared state and the associated complexity. Remoc extends this programming model to distributed systems by providing channels that work seamlessly over remote connections.

For that it uses Serde and the Postbag binary codec to serialize and deserialize data as it is transmitted over an underlying transport, which might be a TCP network connection, a QUIC stream, a WebSocket, UNIX pipe, or even a serial link. Postbag is designed for protocol evolution, allowing many changes to message types without requiring both endpoints to be upgraded at once.

Opening a new channel is straightforward, just send the sender or receiver half of the new channel over an existing channel, like you would do between local threads and tasks. All channels are multiplexed over the same remote connection, with data being transmitted in chunks to avoid one channel blocking another if a large message is transmitted.

// Most Remoc types, like channel halves, can be part of
// serializable data structures.
#[derive(serde::Serialize, serde::Deserialize)]
struct CountReq {
    up_to: u32,
    seq_tx: rch::mpsc::Sender<u32>,
}

// Sending the sender half opens a new channel to the remote
// endpoint, inside the connection that is already established.
let (seq_tx, mut seq_rx) = rch::mpsc::channel();
tx.send(CountReq { up_to: 4, seq_tx }).await.unwrap();

// The remote endpoint counts up to 4 over the channel we provided.
while let Some(i) = seq_rx.recv().await.unwrap() {
    println!("{i}");
}

See the channel example below for the complete, runnable version, including establishing the connection and the code of the remote endpoint.

Building upon its remote channels, Remoc allows calling of remote functions and closures. Furthermore, a trait can be made remotely callable with automatically generated client and server implementations, resembling a classical remote procedure calling (RPC) model; see the RPC example below. When an RPC method returns another remote object, pipelining can send dependent calls without waiting for each intermediate response.

When to use Remoc

Remoc is a good fit once two or more Rust programs need to interact and you would rather express that interaction as channels, function calls and trait objects than design and maintain a custom wire protocol. The processes can either run on the same machine or talk to each other via the network.

Use Remoc to:

• build distributed applications that exchange live channels and objects, not just messages;
• expose a set of related operations as an ordinary Rust trait and call it from the other endpoint,
• talk to a sandboxed or otherwise isolated child process,
• connect a UI to a backend it does not share memory with, including from Rust code compiled to WebAssembly,
• give a remote endpoint a live, read-only mirror of a collection that keeps changing.

Remoc is not:

• a service mesh — it connects exactly two endpoints over one transport connection that you provide; discovery, load balancing and routing between more than two endpoints are outside its scope,
• a message broker — channels and remote objects live only as long as the connection and the process holding them; nothing is persisted or replayed after a restart,
• a network security layer — Remoc neither encrypts nor authenticates the connection itself, see Security below,
• a cross-language protocol — both endpoints of a connection run Rust code using Remoc.

Getting started

Add Remoc, Tokio, and Serde to an application:

cargo add remoc
cargo add serde --features derive
cargo add tokio --features macros,rt-multi-thread,net

The default Remoc features include channels, remote functions and objects, observable collections, and remote trait calling. Applications that only use part of the API can disable default features; see Crate features.

A Remoc application normally follows these steps:

1. Establish an ordered, reliable transport, such as a TCP or TLS stream.
2. Call Connect::io for a byte stream or Connect::framed for a stream of binary messages.
3. Spawn the returned connection future so it can drive all communication.
4. Use the returned base channel directly, or exchange one initial value with ConnectExt::provide and ConnectExt::consume.
5. Send additional channel halves, RTC clients, or remote objects wherever they are needed.

The channel example below demonstrates the base-channel approach in one process; the rtc module documentation does the same for the remote trait calling approach. For separate client and server crates, see the RTC example. The web RTC example runs a Rust client in the browser over a WebSocket.

Connecting

Remoc implements no transport itself and thus depends on no networking crate; it runs over any byte stream you already have. Pass an AsyncRead and AsyncWrite pair, such as a TCP or TLS connection, to Connect::io, or a Sink and Stream of binary packets, such as a WebSocket, to Connect::framed. Both hand you a base channel, over which all further channels and remote objects are exchanged.

The transports module contains worked examples for TCP, TLS, QUIC, WebSocket, pipes to a child process and aggregated, failure-resilient links.

Choosing channels, functions, objects or traits

Everything in Remoc builds on remote channels; which higher-level building block to reach for depends on how your interaction is shaped:

These combine freely: an RTC method can take or return a channel, and a channel's item can contain a remote object.

Forward and backward compatibility

Distributed systems often require that endpoints running different software versions interact. Remoc therefore uses the Postbag Full codec by default. It includes field and variant identifiers and the encoded length of each value, allowing a receiver to skip data it does not know.

With suitable Serde attributes, Postbag supports common schema changes:

• fields can be added, removed, or reordered; whenever a receiver expects a field that the sender omits, that field needs #[serde(default)],
• enum variants can be added, removed, or reordered; an older receiver needs a #[serde(other)] variant to accept an unknown one,
• fields and variants can be renamed without breaking compatibility when they use stable numbered identifiers such as #[serde(rename = "_0")].

An identifier is part of the protocol: changing it is breaking, and an identifier retired from one field or variant must not later be reused for another. Changes to a field's type and other structural transformations are not automatically compatible.

Use codec::recoverable to confine an incompatible field to its default value, or the versioned module when the old representation must be transformed explicitly.

See codec::Postbag and the Postbag documentation for the complete compatibility table and format limitations.

Distributed tracing

Remote trait and function calls can create tracing spans, one at the caller for the duration of the call and one at the server for processing it. The identity of the caller's span travels with the call, so the span of the server becomes a child of it. If an OpenTelemetry layer is installed on the tracing subscriber, its trace and span ids are used and the spans of both endpoints show up in one distributed trace in your collector. Without it, a random span id is recorded on both spans, so that the log lines of client and server can still be matched.

Crate features

Most functionality of Remoc is gated by crate features. The following features are available:

• serde enables the codec module and implements serialize and deserialize for all configuration and error types.
• rch enables remote channels provided by the rch module.
• rfn enables remote function calling provided by the rfn module.
• robj enables remote object utilities provided by the robj module.
• robs enables remotely observable collections provided by the robs module.
• rtc enables remote trait calling provided by the rtc module.
• otel links the spans of remote calls into OpenTelemetry traces, see Distributed tracing.

The meta-feature full enables all features from above but no additional codecs. By default the full feature is enabled.

Data formats for transmission (codecs)

The following features enable additional data formats (codecs) for transmission:

• codec-bincode provides the Bincode 1 and 2 formats
• codec-ciborium provides the CBOR format
• codec-json provides the JSON format
• codec-message-pack provides the MessagePack format
• codec-postcard provides the Postcard format

The feature full-codecs enables all additional data formats.

Remoc uses the full Postbag codec by default, and this is the recommended choice for most applications because it combines compact binary encoding with schema evolution support. Alternative codecs are available for specialized requirements, such as interoperating with an existing format or making a different size, human-readability, or performance trade-off. Choosing one also changes the protocol's compatibility properties, so review that codec's documentation before doing so.

JavaScript and web support

Remoc supports compiling to the WebAssembly targets wasm32-unknown-unknown, wasm32-wasip1 and wasm32-wasip1-threads. If you are targeting a JavaScript runtime environment (like a web browser) you must enable the js crate feature. This will enable JavaScript promises support and spawn tasks onto the browser's native event queue.

Compile time

The explicit-auto-traits feature implements Send and Sync explicitly for most types instead of leaving them to the compiler's auto trait inference. This significantly reduces trait solver recursion and thus compile time; however, it requires allowing unsafe code.

Example

The following example shows a complete Remoc application: two endpoints connected over TCP that exchange values over channels. It is followed by a look at remote procedure calls, the other common style of using Remoc.

Channels

In the following example the server listens on TCP port 9870 and the client connects to it. Then both ends establish a Remoc connection using Connect::io() over the TCP connection. The connection dispatchers are spawned onto new tasks and the client() and server() functions are called with the established base channel. Then, the client creates a new remote MPSC channel and sends it inside a count request to the server. The server receives the count request and counts on the provided channel. The client receives each counted number over the new channel.

use std::net::Ipv4Addr;
use tokio::net::{TcpStream, TcpListener};
use remoc::prelude::*;

#[tokio::main]
async fn main() {
    // For demonstration we run both client and server in
    // the same process. In real life connect_client() and
    // connect_server() would run on different machines.
    tokio::join!(connect_client(), connect_server());
}

// This would be run on the client.
// It establishes a Remoc connection over TCP to the server.
async fn connect_client() {
    // Wait for server to be ready.
    tokio::time::sleep(std::time::Duration::from_secs(1)).await;

    // Establish TCP connection.
    let socket = TcpStream::connect((Ipv4Addr::LOCALHOST, 9870))
        .await
        .unwrap();
    let (socket_rx, socket_tx) = socket.into_split();

    // Establish Remoc connection over TCP.
    // The connection is always bidirectional, but we can just drop
    // the unneeded receiver.
    let cfg = remoc::Cfg::default();
    let (conn, tx, _rx): (_, _, rch::base::Receiver<()>) =
        remoc::Connect::io(cfg, socket_rx, socket_tx)
            .await
            .unwrap();
    tokio::spawn(conn);

    // Run client.
    client(tx).await;
}

// This would be run on the server.
// It accepts a Remoc connection over TCP from the client.
async fn connect_server() {
    // Listen for incoming TCP connection.
    let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 9870))
        .await
        .unwrap();
    let (socket, _) = listener.accept().await.unwrap();
    let (socket_rx, socket_tx) = socket.into_split();

    // Establish Remoc connection over TCP.
    // The connection is always bidirectional, but we can just drop
    // the unneeded sender.
    let cfg = remoc::Cfg::default();
    let (conn, _tx, rx): (_, rch::base::Sender<()>, _) =
        remoc::Connect::io(cfg, socket_rx, socket_tx)
            .await
            .unwrap();
    tokio::spawn(conn);

    // Run server.
    server(rx).await;
}

// User-defined data structures needs to implement Serialize
// and Deserialize.
#[derive(Debug, serde::Serialize, serde::Deserialize)]
struct CountReq {
    up_to: u32,
    // Most Remoc types like channels can be included in
    // serializable data structures for transmission to remote
    // endpoints.
    seq_tx: rch::mpsc::Sender<u32>,
}

// This would be run on the client.
// It sends a count request to the server and receives each number
// as it is counted over a newly established MPSC channel.
async fn client(mut tx: rch::base::Sender<CountReq>) {
    // By sending seq_tx over an existing remote channel, a new
    // remote channel is automatically created and connected to the
    // server. This all happens inside the existing TCP connection.
    let (seq_tx, mut seq_rx) = rch::mpsc::channel();
    tx.send(CountReq { up_to: 4, seq_tx }).await.unwrap();

    // Receive counted numbers over new channel.
    assert_eq!(seq_rx.recv().await.unwrap(), Some(0));
    assert_eq!(seq_rx.recv().await.unwrap(), Some(1));
    assert_eq!(seq_rx.recv().await.unwrap(), Some(2));
    assert_eq!(seq_rx.recv().await.unwrap(), Some(3));
    assert_eq!(seq_rx.recv().await.unwrap(), None);
}

// This would be run on the server.
// It receives a count request from the client and sends each
// number as it is counted over the MPSC channel sender provided
// by the client.
async fn server(mut rx: rch::base::Receiver<CountReq>) {
    // Receive count request and channel sender to use for counting.
    while let Some(CountReq { up_to, seq_tx }) =
        rx.recv().await.unwrap()
    {
        for i in 0..up_to {
            // Send each counted number over provided channel.
            seq_tx.send(i).await.unwrap();
        }
    }
}

Remote procedure calls

Channels are the foundation, but a remote endpoint that should expose several related methods is usually better served by remote trait calling (RTC). Tagging a trait generates a client that implements it and servers that execute the calls on your object, resembling a classical RPC model — while remaining free to pass channels and remote objects through the calls:

use remoc::prelude::*;
use remoc::rtc::CallError;

// Tagging the trait generates CounterClient and the
// CounterServer* types.
#[rtc::remote]
pub trait Counter {
    async fn value(&self) -> Result<u32, CallError>;

    async fn increase(&mut self, by: u32) -> Result<(), CallError>;

    // Methods can take and return channels and other remote
    // objects.
    async fn watch(
        &mut self,
    ) -> Result<rch::watch::Receiver<u32>, CallError>;
}

// CounterClient implements Counter, so calling it looks like a
// local call, but is executed on the counter object located on
// the server.
async fn use_counter(
    mut counter: CounterClient,
) -> Result<(), CallError> {
    counter.increase(5).await?;
    assert_eq!(counter.value().await?, 5);

    // The watch receiver returned by the call stays connected to
    // the counter object and reports every change made to it.
    let watch_rx = counter.watch().await?;
    assert_eq!(*watch_rx.borrow().unwrap(), 5);

    Ok(())
}

The client is remote sendable, so it can be sent over any channel, just like the channel halves above, or transferred while establishing the connection using ConnectExt::provide and ConnectExt::consume. See the rtc module documentation for the server side, connecting and a complete example, and the RTC example for client and server split into separate crates.

Security

Remoc neither encrypts nor authenticates the connection; it is designed to run on top of a transport that already provides the properties you need. If a connection crosses a trust boundary, wrap the transport in TLS or another secure channel — see the TLS transport example — before passing it to Connect::io or Connect::framed.

When exchanging data with an untrusted or unauthenticated endpoint, also review the size considerations in the remote channel module and the max_ports and connect_queue settings of Cfg, which bound how many channels a peer can make you open.

Supported Rust versions

Remoc is built against the latest stable release. The minimum supported Rust version (MSRV) is 1.95.

Development

Development on native platforms is straightforward. Use cargo test to run tests as usual.

To run tests in a JavaScript runtime environment (for example wasm32-unknown-unknown with js feature) install wasm-bindgen-test-runner and Google ChromeDriver. Then use the following command to execute the test suite:

WASM_BINDGEN_USE_BROWSER=1 WASM_BINDGEN_TEST_TIMEOUT=90 \
    cargo +nightly test --target wasm32-unknown-unknown \
    --all-features --release --tests

A proper web-compatible runtime environment is required. Thus Node.js will not work. Deno should work, but it currently has some issues with the interaction between WebAssembly and async execution.

Sponsors

Development of Remoc is partially sponsored by ENQT GmbH and mlilabs GmbH.

License

Remoc is licensed under the Apache 2.0 license.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Remoc by you, shall be licensed as Apache 2.0, without any additional terms or conditions.