This repository contains a Java-based distributed SQL execution engine. It is implemented as a pedagogical proof-of-concept demonstrating distributed query planning, sharded execution, and streaming Volcano-style iterators over gRPC.
- Parses a limited subset of SQL using
JSQLParser. - Generates logical and physical execution plans.
- Routes queries based on exact equality matches against a statically configured
shard_key. - Executes distributed queries across mock or PostgreSQL-backed worker nodes.
- Streams results incrementally over gRPC.
The implementation is strictly limited in scope. It does not support:
- High Availability: The Coordinator is a single point of failure (SPOF). There is no leader election, Raft/Paxos consensus, or state replication implemented.
- Cross-Shard Joins: Distributed shuffle joins and broadcast joins are not implemented in the physical execution engine.
- External Sorting: The
SortOperatorrequires full memory materialization. It does not spill to disk. - DML/DDL:
INSERT,UPDATE,DELETE, and schema modification commands are unsupported. - Complex Subqueries: CTEs and nested subqueries are not supported.
- Security (Plaintext Only): All gRPC and JDBC connections operate in plaintext. No TLS, mTLS, or Authentication is implemented. This engine must not be deployed over untrusted networks.
- Simplicity over Resilience: By relying on a single Coordinator, the network topology is drastically simplified, but the entire cluster becomes unavailable if the Coordinator process crashes.
- Pushdown vs. Network Transfer: The system aggressively attempts to push filters down to the PostgreSQL worker nodes to minimize network serialization overhead, at the cost of requiring the underlying workers to possess robust indexing.
- Memory: Coordinator memory during aggregations scales
O(K)(whereKis the number of distinct groups). Streaming operators use bounded queues (e.g., 10,000 items). Sorting is unboundedO(N). - Failures: Transient network failures trigger bounded exponential retries. Persistent worker failures result in immediate query abortion. The engine explicitly rejects partial results to maintain mathematical consistency.
common/: Shared Protobuf definitions (QueryProto.proto).coordinator/: Parsing, planning, and the Volcano execution engine.worker/: PostgreSQL JDBC execution and gRPC streaming services.client/: Simple CLI submission interface.backend//frontend/: Experimental web visualization (not critical path).