Repository navigation
[Tracking Issue] Integration with big data ecosystem #5905
Description
What type of enhancement is this?
User experience
What does the enhancement do?
GreptimeDB, a hybrid time series and analytical database, can manage real-time data ingestion and analytics independently. However, for many established infrastructures, integrating with existing big data systems such as Flink, Kafka, BigQuery, Hadoop, and Spark is essential. This issue focuses on the integration process within the big data ecosystem.
To clarify further, we can categorize these integrations into two parts: GreptimeDB as a data sink and GreptimeDB as a data source.
GreptimeDB as data sink
This direction aims to ingest data from big data softwares to faciliate fast, low cost and efficient analytics.
- Elasticsearch exporter: Export logs in Elasticsearch to GreptimeDB in an efficient way.
- Flink sink: Set up sinks to write streaming data to GreptimeDB.
- Kafka connector: Use Kafka Connect or custom sink connectors to send data to GreptimeDB.
GreptimeDB as data source
- Kafka: Use change data capture (CDC) connectors to stream GreptimeDB updates to Kafka.
Implementation challenges
No response
Activity
integral-830 commentedon Oct 15, 2026
Hi @v0y4g3r , I'd like to take the Kafka to GreptimeDB sink item, implemented in Rust rather than as a Kafka Connect connector (Connect is JVM-only; the issue also allows "custom sink connectors").
What I found while scoping:
Proposal: a standalone service built on
rdkafkaandgreptimedb-ingester. It consumes topics (JSON first, then Avro/Protobuf), maps them to tables via config (time index, tags, fields), batches rows, writes through the simple or bulk-stream path, and commits offsets only after a successful write (at-least-once). Failed records go to a DLQ, and it exposes Prometheus metrics.Questions:
GreptimeTeamrepo or elsewhere?