d-engine: A Lightweight Distributed Coordination Engine for Rust

Source: Dev.to

Overview

A lightweight Raft implementation designed for embedding into Rust applications — the consensus layer for building reliable distributed systems.
Built with a simple vision: make distributed coordination accessible, cheap to run, and simple to use.
Core philosophy: prefer simple architectures over complex ones.

• Single‑threaded event loop – no race conditions, strict ordering; CPU‑bound on a single core, scale horizontally.
• Role separation (SRP) – Leader, Follower, Candidate, Learner each handle their own logic; the main loop just routes events.
• Standard Raft design – follows the typical Raft architecture.

Modes of Operation

Embedded mode

Runs inside your Rust process.

let engine = EmbeddedEngine::start().await?;
engine.wait_ready(Duration::from_secs(5)).await?;

let client = engine.client();
client.put(b"key".to_vec(), b"value".to_vec()).await?; // Lab numbers only—production performance varies by workload and hardware.

Pluggable Architecture

d-engine provides working implementations (RocksDB storage, KV operations, gRPC transport). When defaults don’t fit, implement the required traits:

pub trait StorageEngine {
    type LogStore: LogStore;
    type MetaStore: MetaStore;
}

pub trait StateMachine {
    async fn apply_chunk(&self, entries: Vec) -> Result;
}

Examples in the repository demonstrate:

• Sled storage backend.
• Custom HTTP handlers with HAProxy for high‑availability deployments.

Consistency Guarantees

Three read policies (configurable per operation or server‑wide):

Defaults are sane; trade‑offs are documented.

Scaling Example

// Start with 1 node (auto‑elected leader)
let engine = EmbeddedEngine::start().await?;

// Scale to 3 nodes later: update config, zero code changes
// See `examples/single-node-expansion`

No Kubernetes or complex setup required—just Rust + a configuration file.

Features Summary

• Raft consensus implementation.
• Pluggable storage (RocksDB, Sled, custom).
• Flexible consistency (linearizable, lease, eventual).
• Production‑ready core; API stabilizing toward v1.0.
• Version 0.2: core engine is production‑ready (1000+ tests, TLA+ & Jepsen validated).

Future Direction

• Cloud‑native deployment (Cloudflare, AWS, GCP storage backends). Timeline depends on real‑world use cases from early adopters.
• Exploring an etcd‑compatible API layer.

Getting Started

[dependencies]
d-engine = "0.2"

• GitHub:
• Documentation:
• Examples:

If you’re building distributed systems in Rust and need consensus, the code is ready. The repository includes examples for embedded mode, standalone mode, custom storage, and HA deployments with HAProxy.

Call for Users

We’re looking for Rust developers building distributed systems who have real problems to solve:

• Coordination bottlenecks (slow consensus, expensive etcd clusters).
• Strong consistency requirements (leader election, distributed locks, metadata stores).
• Production deployments needing cheap, simple, reliable coordination.

Ideal scenarios

• Production‑grade deployments (not toy projects).
• Cloud‑native environments (Cloudflare Workers, AWS Lambda, serverless patterns).
• Cost‑sensitive use cases where etcd is overkill.

If you have a specific problem, open an issue with your use case or reach out directly.

License & Compatibility

• License: MIT or Apache‑2.0
• Platforms: Linux, macOS
• Minimum Supported Rust Version (MSRV): 1.88