A high-performance C++20 port of the Narwhal and Tusk DAG-based mempool and consensus engine, featuring modular support for Shoal++ and Mysticeti
This project is designed as a modular, high-fidelity implementation of state-of-the-art consensus protocols, focusing on low latency and horizontal scalability.
The primary goal of this project is to evaluate the performance of the Narwhal++ mempool implementation in C++20 when integrated with various state-of-the-art BFT consensus engines.
By providing a high-fidelity C++ port, this repository enables a direct performance comparison between:
- Tusk: The baseline DAG-based consensus.
- Shoal++: Performance-optimized selection with reputation.
- Mysticeti: The latest advancement in low-latency uncertified DAG consensus.
This evaluation focuses on identifying bottlenecks, measuring throughput (certificates/sec), and analyzing latency characteristics across different consensus strategies within a unified C++20 framework.
[ Clients ]
│
▼
┌───────────────────────┐ ┌─────────────────────────┐
│ Narwhal Mempool │ │ Consensus Engines │
│ (Data Availability) │ │ (Total Ordering) │
├───────────────────────┤ ├─────────────────────────┤
│ ┌───┐ ┌───┐ ┌───┐ │ │ ┌───────────────┐ │
│ │ B │──▶│ B │──▶│ B │ │ │ │ Mysticeti │ │
│ └───┘ └───┘ └───┘ │ │ └───────────────┘ │
│ ▲ ▲ ▲ │ │ ▲ │
│ │ │ │ ◀──────┼──────────┘ │
│ ┌───┐ ┌───┐ ┌───┐ │ │ ┌───────────────┐ │
│ │ C │──▶│ C │──▶│ C │ │ │ │ Shoal++ │ │
│ └───┘ └───┘ └───┘ │ │ └───────────────┘ │
│ ▲ ▲ ▲ │ │ ▲ │
│ │ │ │ ◀──────┼──────────┘ │
│ ┌───┐ ┌───┐ ┌───┐ │ │ ┌───────────────┐ │
│ │ G │──▶│ G │──▶│ G │ │ │ │ Tusk │ │
│ └───┘ └───┘ └───┘ │ │ └───────────────┘ │
└───────────────────────┘ └─────────────────────────┘
Shared Mempool DAG Modular Pluggable
(C++20 Buffers) Logic Layers
- Modular Consensus Engines: Switch seamlessly between:
- Tusk: High-throughput DAG-based consensus.
- Shoal++: Advanced anchor selection with reputation systems.
- Mysticeti: Ultra-low latency "3-hop" consensus path.
- DAG-based Architecture: Decouples data availability from transaction ordering.
- High-Performance C++20: Utilizing modern C++ features for maximum efficiency.
- Windows-Friendly: Integrated PowerShell scripts for local cluster simulation and benchmarking.
- Pluggable Backend: Support for both production-ready dependencies (RocksDB, Sodium, Boost.Asio) and internal mocks for rapid testing/CI.
- Core: C++20
- Build System: CMake 3.20+
- Asynchronous I/O: Boost.Asio
- Storage: RocksDB
- Cryptography: Libsodium (Ed25519)
- Networking: TLS 1.3 / Noise Protocol
- A modern C++ compiler (MSVC 2022+ or GCC 11+)
- CMake 3.20 or higher
# Create build directory and configure
cmake -B build -S .
# Build in Release mode
cmake --build build --config ReleaseLaunch a 4-node local cluster with the Mysticeti engine:
./run_cluster.ps1 -Engine mysticeti -LoadPossible engines: tusk, shoal++, mysticeti.
These benchmarks were performed on a local 4-node cluster using internal mocks for networking and storage to measure the maximum theoretical throughput of the consensus engines.
| Engine | Throughput (Cert/sec) | Latency (ms) | Key Mechanism |
|---|---|---|---|
| Tusk | ~19,800 | ~450 | DAG-link based commit |
| Shoal++ | ~22,600 | ~320 | Adaptive Anchor selection + Reputation |
| Mysticeti | ~2,150 (Leaders) | ~120 | 3-hop Fast Path (Fastest Finality) |
Note: Tusk and Shoal++ report throughput of all committed certificates in the DAG, while Mysticeti throughput focuses on individual leader-round commitments for ultra-low latency.
Currently Implemented (Real):
- ✅ Core DAG consensus logic for all three engines
- ✅ Certificate creation, validation, and ordering
- ✅ Multi-threaded consensus execution
- ✅ Modular engine architecture
Currently Mocked (For Testing):
- 🔶 Network layer (TLS 1.3 connections simulated)
- 🔶 Cryptographic signatures (Ed25519 placeholder)
- 🔶 Persistent storage (In-memory maps instead of RocksDB)
Important: This is a research and evaluation platform, not a production-ready system. The mocks enable rapid prototyping and benchmarking of consensus algorithms without the overhead of full network I/O.
Given the critical nature of BFT consensus correctness, we are planning a multi-layered verification approach:
-
Formal Verification (Planned):
- TLA+ specifications for consensus invariants
- Property-based testing with RapidCheck (C++ equivalent to Rust's proptest)
- Model checking for liveness and safety properties
-
Byzantine Fault Simulation (Phase 4):
- Adversarial node behavior injection
- Network partition and delay simulation
- Equivocation detection testing
-
Observability (Phase 1):
- Prometheus/Grafana metrics integration
- OpenTelemetry distributed tracing
- Consensus state visualization
- Core DAG Logic & Tusk Implementation
- Shoal++ & Mysticeti engines
- Phase 0: Formal Verification Foundation
- TLA+ specifications for consensus safety/liveness
- Property-based testing suite (RapidCheck)
- Invariant checking in CI/CD
- Phase 1: Infrastructure Professionalization
- Asynchronous Multi-threaded Networking (Boost.Asio)
- Real Ed25519 signatures with batch verification
- Prometheus metrics & Grafana dashboards
- OpenTelemetry distributed tracing
- Phase 2: Advanced BFT Liveness
- Pacemaker & Timeout Management
- View-change protocol
- Hot-reloading configuration
- Phase 3: Scalable Data Availability (Separate Worker-Layer)
- Phase 4: Hardening (Byzantine Simulation & Fuzzing)
This project is licensed under the MIT License.