Tower + VeloDB — Ask Your Lakehouse

Stand up a working Tower + VeloDB demo on your own data: Tower lands data in an open Apache Iceberg lakehouse — ideally a Tower Catalog (Tower's own hosted, Polaris-based Iceberg catalog + storage) — VeloDB serves it fast and answers questions in plain English (its native AI_GENERATE writes the SQL). Fill in .env, run one preflight, deploy three small templates.

Tower pipelines ─► Iceberg lakehouse ─mount─► VeloDB catalog ─► native table (fast)
                                                                      ▲
   ask-dashboard ─► AI_GENERATE (LLM) ─► SQL ─────────────────────────┘ ─► results + chart

Using this as an AI skill (recommended)

This repo is packaged as an agent skill (skills/tower-velodb/, same layout as tower/agent-skills). Installed as a skill, an AI agent (Claude Code, Cursor, Tower Control, …) drives the whole flow for you — interviews you for the config, runs the preflight, diagnoses failures against the built-in Troubleshooting table, and deploys the templates.

Install it with the skills CLI (this is the "npm" flow — skills is distributed via npm and run with npx):

npx skills add tower/tower-velodb-skill      # from this repo
# or preview what it would add:
npx skills add tower/tower-velodb-skill --list

That places skills/tower-velodb/ (SKILL.md + its templates/ and scripts/) where your agent discovers it. A skill is found by its SKILL.md at .../skills/<name>/SKILL.md; the supporting files sit alongside and are referenced by relative paths, so they resolve after install.

Invoke it — in your agent, name the skill or just describe the goal:

/tower-velodb

or simply ask:

"Use the tower-velodb skill to serve our lakehouse through VeloDB and let us query it in plain English."

The agent reads SKILL.md, asks for the VeloDB / catalog / LLM details, writes your .env, runs the preflight, and stands up the templates — pausing on any failure with the specific fix from the Troubleshooting table.

Prefer to do it by hand? You don't need the skill — the templates are plain, self-contained apps. Follow Quick start (local) below (run from inside skills/tower-velodb/).

What's here

Everything lives under skills/tower-velodb/:

• SKILL.md — the playbook: architecture, setup interview, Troubleshooting table, and "what this skill is not for" (read this when anything breaks).
• scripts/setup_and_preflight.py — one-time setup + validation. Run first.
• scripts/perf_compare.py — Iceberg-direct vs native timing on your data.
• templates/refresh_to_velodb — materialize Iceberg → fast native VeloDB table.
• templates/ask_dashboard — the ask-your-lakehouse Marimo UI.
• templates/populate_demo_data — optional synthetic data generator (if you don't yet have a table to serve).
• templates/lib/velodb.py — shared config + connection + the robust patterns (catalog refresh/retry, AI-resource setup, schema introspection).

Prerequisites

• A VeloDB Cloud cluster (or Doris 4.0+): host, port 9030, user, password.
• An Iceberg lakehouse via an Iceberg REST catalog. Easiest option: a Tower Catalog — Tower's own hosted, Apache Polaris-based Iceberg catalog and storage. Tower provisions the catalog + object storage for you and injects the connection details into your Tower apps automatically, so there's nothing to stand up. Any other Iceberg REST catalog also works (Snowflake Open Catalog, Unity, Glue-REST) — you just supply the URI, warehouse, OAuth2 credential, scope, and region yourself.
• A table already in that lakehouse to serve (or use populate_demo_data).
• An LLM API key for AI_GENERATE (Anthropic / OpenAI / DeepSeek / local). The account must have credit.
• uv and the tower CLI (tower login done).

Quick start (local)

cd skills/tower-velodb        # all paths below are relative to the skill dir
cp .env.example .env          # fill in VELODB_*, ICEBERG_*, AI_*
set -a; . ./.env; set +a

# 1. Create catalog + AI resource and validate everything. Fix issues here.
uv run --with pymysql python scripts/setup_and_preflight.py

# 2. Materialize Iceberg -> fast native table.
cd templates/refresh_to_velodb && uv run --with pymysql python main.py && cd -

# 3. Run the dashboard.
cd templates/ask_dashboard && uv run python main.py     # http://localhost:50051

Running on Tower

The templates read config from the environment, so the same code runs locally (from .env) or on Tower (from secrets). To deploy:

1. Create the same values as Tower secrets (tower secrets create --name VELODB_HOST --value ... for each VELODB_*, ICEBERG_*, AI_*). Do not commit .env.
2. cd templates/refresh_to_velodb && tower deploy -f then schedule it: tower schedules create --app refresh-to-velodb --cron "*/5 * * * *"
3. cd templates/ask_dashboard && tower deploy -f && tower run (long-running web app).

Security

.env is gitignored; only .env.example (placeholders) is committed. Never paste real keys into chat or commit them. Rotate any key that leaks. AI_API_KEY and VELODB_PASSWORD are the sensitive ones.

The gotchas (short list — full detail in skill/SKILL.md)

• Catalog needs vended-credentials-enabled='true' + client.region (not s3.region); Polaris also needs nested-namespace-enabled='true'.
• Vended S3 creds expire (~1h) → REFRESH CATALOG + retry (built into the lib).
• SET default_ai_resource is session-scoped.
• VeloDB can't expire Iceberg snapshots → writers must commit infrequently or the 5MB metadata ceiling wedges the table.
• Bedrock isn't usable for AI_GENERATE (SigV4); use a static-key provider.