feat(observability): add Prometheus metrics, structured logging, OpenAPI, compose & k8s manifests

[ashkantaravati]

Motivation

• Provide operational visibility (metrics + structured logs) for job lifecycle, HTTP traffic, queue/storage state and render performance so ChromaFlow is easier to monitor and operate.
• Add local observability and deployment artifacts (Docker Compose + Kubernetes) and basic API docs to simplify running and integrating the service and to prepare for future Redis/RabbitMQ/Kafka/MinIO/auth additions.

Description

• Add a new internal/observability package with Metrics (Prometheus text exposition writer, HTTP middleware, histogram buckets, queue/storage hooks) and NewLogger (structured JSON log/slog) and expose /metrics via a handler.
• Instrument HTTP handlers and worker lifecycle to emit structured JSON logs and metrics (job submitted/rejected/finished, render durations, PDF bytes, active workers, queue depth/capacity, storage counts, and HTTP request histograms).
• Serve an OpenAPI 3.0 YAML document at GET /openapi.yaml describing the public endpoints and schemas.
• Extend in-memory queue/storage with Len(), Cap() and Stats() helpers so metrics can report queue depth and storage breakdown.
• Add a Docker Compose observability stack and provisioning files for Prometheus, Grafana, Loki and Promtail with sensible defaults and a bundled Grafana dashboard and Prometheus scrape config.
• Add plain Kubernetes manifests in k8s/ to deploy ChromaFlow plus Prometheus/Grafana/Loki/Promtail (Ingress, Services, Deployments/DaemonSet, ConfigMaps and notes).
• Update README.md, AGENTS.md, IMPROVEMENTS.md, Makefile and tests to document and wire the new features and to add compose-up/k8s-apply targets.

Testing

• Ran formatting and style checks with gofmt -w and the repo check test -z "$(gofmt -l $(git ls-files '*.go'))" which succeeded.
• Ran unit/handler tests with GOCACHE=/tmp/chromaflow-gocache go test ./... which passed for the repo packages.
• Ran vet with GOCACHE=/tmp/chromaflow-gocache go vet ./... which completed without findings.
• Built the binary with GOCACHE=/tmp/chromaflow-gocache CGO_ENABLED=0 go build -trimpath -o /tmp/chromaflow-build ./cmd/server which succeeded.
• Validated Kubernetes and Compose YAML with local parsers (ruby YAML load) which confirmed the files parse; docker compose config could not be run here because Docker is not installed in this environment.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f02c746d4a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Collapse unmatched request paths before recording metrics

For requests that do not match a ServeMux route, such as random 404 probes or method-mismatch 405s like POST /pdf/<id>, r.Pattern is empty, so this fallback records the raw URL path as a Prometheus label. Because RecordHTTP stores each label combination in maps for the lifetime of the process, arbitrary unique paths can create unbounded metric cardinality and memory growth; use a fixed label like unknown/not_found for empty patterns instead.

Useful? React with 👍 / 👎.