Ticket Marketplace Analytics Engine

A GDPR-compliant, horizontally scalable real-time analytics engine for ticket marketplace platforms. Built with Kafka, Flink, ClickHouse, dbt, and Metabase.

Architecture

Ticket Events → Kafka → Flink → ClickHouse → dbt → Metabase
(Vendors/Customers)

Components:

• Kafka: Message streaming platform for ticket events
• Flink: Real-time stream processing for ticket transactions
• ClickHouse: Columnar OLAP database for analytics
• dbt: Data transformation and business metrics
• Metabase: Data visualization for dashboards

Prerequisites

• Docker & Docker Compose installed
• At least 8GB RAM available for Docker
• 20GB free disk space
• Python 3.8+ (for dbt and event simulators)

Quick Start

1. Start All Services

# Navigate to project root
cd <project-root>

# Start all containers
docker-compose up -d

# Wait 30-60 seconds for services to initialize
# Check status
docker-compose ps

Kafka topics are automatically created by the kafka-init container.

2. Submit Flink Job

# Submit the Flink streaming job (Scala JAR)
python scripts/start_flink_job.py

This submits the Scala streaming job that processes events from Kafka to ClickHouse.

3. Install Python Dependencies

# Create virtual environment (if not exists)
python -m venv .venv

# Activate virtual environment
# Windows PowerShell:
.venv\Scripts\Activate.ps1
# Linux/Mac:
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt
pip install -r tests/requirements.txt

4. Run dbt Transformations

cd dbt

# Install dbt and ClickHouse adapter
pip install "dbt-core>=1.7,<2.0" "dbt-clickhouse>=1.5,<2.0"

# Test connection
dbt debug

# Build all models
dbt run

# Run tests
dbt test

5. Bootstrap Metabase Dashboards

# Install Metabase bootstrap dependencies
pip install requests

# Run bootstrap script (wait 60 seconds after docker-compose up)
python scripts/bootstrap_metabase.py

This creates admin user, ClickHouse connection, and 5 dashboards.

6. (Optional) Generate Test Data

cd simulators
pip install -r requirements.txt

# Run simulator with default settings
python ticket_marketplace_simulator.py

# Or run a specific scenario
python ticket_marketplace_simulator.py --scenario flash_sale

7. Access Dashboards

• Metabase: http://localhost:3000 (login: admin@example.com / admin123)
• Kafka UI: http://localhost:8080
• Flink Dashboard: http://localhost:8081
• ClickHouse: http://localhost:8123/play

cd simulators
pip install -r requirements.txt

# Run default simulation (10 minutes)
python ticket_marketplace_simulator.py

# Run flash sale scenario (high traffic)
python ticket_marketplace_simulator.py --scenario flash_sale

# Custom configuration
python ticket_marketplace_simulator.py \
  --vendors 100 \
  --customers 5000 \
  --duration 30 \
  --purchases-per-min 200

5. Run dbt Transformations

cd dbt

# Install dbt-clickhouse
pip install dbt-clickhouse

# Test connection
dbt debug

# Run all models
dbt run

# Run specific model
dbt run --select vendor_performance

# Run tests
dbt test

# Generate documentation
dbt docs generate
dbt docs serve  # Opens in browser (default port 8080, may conflict with Kafka UI)

Project Structure

ticket-analytics-engine/
├── docker-compose.yml          # Docker Compose configuration
├── .env                        # Environment variables
├── README.md                   # This file
├── flink/
│   ├── Dockerfile              # Flink custom image (builds Scala job)
│   ├── job/                    # Scala project (Maven)
│   │   ├── pom.xml
│   │   └── src/main/scala/com/ibook/flink/
│   │       └── KafkaToClickHouseJob.scala  # Scala streaming job
│   └── jobs/
│       └── job_contract.yaml  # Job configuration (mounted at /opt/flink/jobs/)
├── clickhouse/
│   ├── config.xml              # ClickHouse server config
│   └── init/
│       └── 01_init_tables.sql  # Database schema and tables
├── dbt/
│   ├── dbt_project.yml         # dbt project configuration
│   ├── profiles.yml             # Connection profiles
│   └── models/
│       ├── schema.yml          # Model documentation and tests
│       ├── staging/            # Staging models
│       └── marts/              # Mart models
├── simulators/
│   ├── requirements.txt        # Python dependencies
│   ├── config.yaml             # Simulation configuration
│   ├── ticket_marketplace_simulator.py  # Event simulator
│   └── README.md               # Simulator documentation
├── specs/                      # Detailed specifications
│   ├── kafka-spec.md
│   ├── flink-spec.md
│   ├── clickhouse-spec.md
│   ├── dbt-spec.md
│   ├── metabase-spec.md
│   ├── simulator-spec.md
│   ├── gdpr-compliance-spec.md
│   └── infrastructure-spec.md
└── docs/
    └── TODO/
        └── implementation_plan.md  # Phased implementation plan

Key Features

Real-Time Processing

• Stream processing with Apache Flink
• Low-latency event ingestion
• Real-time aggregations via ClickHouse materialized views

GDPR Compliance

• User consent filtering at stream level
• 90-day data retention policy
• Right to access and right to erasure support
• Audit logging for data access

Scalability

• Horizontal scaling for Kafka, Flink, and ClickHouse
• Partitioned tables for efficient queries
• Materialized views for pre-aggregated metrics

Analytics

• Vendor performance metrics
• Ticket sales analytics
• Customer behavior segmentation
• Conversion funnel analysis

Monitoring

Check ClickHouse Data

# Check data ingestion
docker exec -it clickhouse clickhouse-client --query "
SELECT count() as total_purchases 
FROM ticket_analytics.ticket_purchases_raw;
"

# View real-time aggregations
docker exec -it clickhouse clickhouse-client --query "
SELECT 
    toStartOfHour(purchase_timestamp) as hour,
    count() as purchases,
    sum(total_price) as revenue
FROM ticket_analytics.ticket_purchases_raw
GROUP BY hour
ORDER BY hour DESC
LIMIT 24;
"

Monitor Kafka Topics

# Monitor Kafka topics
docker exec -it kafka kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic ticket-purchases \
  --from-beginning \
  --max-messages 10

Documentation

• Setup Guide: docs/how_to_run.md - Complete setup instructions
• Testing Guide: docs/how_to_test.md - Testing scenarios and validation
• How It Works: docs/how_it_works.md - Component explanations and algorithms
• Specifications: See specs/ directory for detailed module specifications
• Implementation Plan: See docs/TODO/implementation_plan.md for phased development plan

Common Commands

# Start services
docker-compose up -d

# Stop services
docker-compose down

# Stop and remove volumes (WARNING: deletes all data)
docker-compose down -v

# View logs
docker-compose logs -f [service-name]

# Scale Flink TaskManagers
docker-compose up -d --scale flink-taskmanager=4

# Access ClickHouse CLI
docker exec -it clickhouse clickhouse-client

# Access Kafka CLI
docker exec -it kafka bash

Troubleshooting

Services Not Starting

• Check Docker has enough resources (8GB+ RAM)
• Verify ports are not in use
• Check logs: docker-compose logs [service-name]

Flink Job Not Processing

• Verify Kafka topics exist
• Check Flink UI: http://localhost:8081
• Verify ClickHouse is accessible from Flink

No Data in ClickHouse

• Verify Flink job is running
• Check Kafka has messages
• Verify GDPR filtering (only consented users)

Next Steps

1. Review specifications in specs/ directory
2. Follow implementation plan in docs/TODO/implementation_plan.md
3. Customize configurations for your use case
4. Set up monitoring and alerting
5. Configure production deployment

License

MIT License - Feel free to modify and use for your projects.