A big data project to develop a real-time data pipeline for analyzing the popularity and sentiments of trending topics on Twitter.
-
Updated
Jun 21, 2022 - Scala
A big data project to develop a real-time data pipeline for analyzing the popularity and sentiments of trending topics on Twitter.
A high-throughput, real-time Medallion Lakehouse platform processing mock financial transactions using Apache Flink, Kafka KRaft, Apache Iceberg, Trino, and dbt with active state management and PII hashing.
Low-latency real-time fleet telemetry pipeline. Ingests, validates, and statefully aggregates (sliding event-time windows) vehicle diagnostics under 148ms. Features Kafka buffers, PyFlink stream-processing, S3 checkpoints, and Redis active cache serving. Handles 50k events/sec with custom watermark handling, and schema validation gates.
Add a description, image, and links to the realtime-data-pipeline topic page so that developers can more easily learn about it.
To associate your repository with the realtime-data-pipeline topic, visit your repo's landing page and select "manage topics."