feat: migrate to V4 binary topic format

[neekolas]

Summary

Migrate the notification server from V3 string-based topic identifiers (e.g., /xmtp/mls/1/g-{hex}/proto) to V4 binary topics as defined by xmtpd/pkg/topic. This enables support for V4 clients while maintaining full backwards compatibility for existing V3 consumers.

Stack: This PR is stacked on #93 (Go 1.26 upgrade), which is a prerequisite for the xmtpd dependency.

What changed

New: Topic conversion module (pkg/topics/)

• ParseV3Topic() — converts V3 string topics to *topic.Topic
• TopicToString() — converts *topic.Topic back to V3 string representation
• GetMessageTypeFromTopic() — extracts message type from binary topic kind
• Removed dead code: GetMessageType(), messageTypeByPrefix, HmacUpdates

Proto: New binary topic fields

• Subscription.topic_bytes (field 4) — binary topic, takes precedence over string topic
• SubscribeRequest.topics_bytes (field 3) — merged with string topics
• UnsubscribeRequest.topics_bytes (field 3) — merged with string topics
• Existing field numbers are unchanged — fully wire-compatible

Database: TEXT → BYTEA migration

• Migration 00003: Renames topic to topic_legacy, adds topic BYTEA, converts conforming V3 topics inline using lower() for hex normalization, deletes non-conforming rows, deduplicates collapsed case variants, recreates indexes
• Migration 00004: Drops topic_legacy column
• SQLC schema and queries updated for BYTEA parameters

Core interfaces (pkg/interfaces/)

• Subscription.Topic changed from string to *topic.Topic
• Added Subscription.TopicString for backwards-compatible JSON serialization
• Custom MarshalJSON emits both "topic" (string) and "topicBytesB64" (base64)
• Subscriptions interface methods now accept []*topic.Topic

API layer (pkg/api/)

• normalizeTopics() merges string + binary topic inputs with deduplication
• normalizeSubscriptionInputs() handles SubscribeWithMetadata with bytes-takes-precedence logic
• Invalid topics return InvalidArgument with descriptive error messages

Listener (pkg/xmtp/)

• processEnvelope() converts V3 string topics to binary before subscription lookup
• Fast-path strings.HasPrefix check before expensive parsing
• Non-convertible topics logged at Info level and dropped (not an error)
• getContext() now uses GetMessageTypeFromTopic() for message classification

Delivery services (pkg/delivery/)

• APNS and FCM payloads include both "topic" (reconstructed V3 string) and "topicBytesB64" (base64-encoded binary)
• HTTP delivery gets both fields automatically via Subscription.MarshalJSON()

Important design considerations

Intentional compatibility break for non-conforming topics

Non-conforming topics (e.g., non-hex identifiers like test_installation) are deleted from the database during migration and rejected by the API going forward. This is acceptable because non-conforming topics cannot be delivered on the V4 network. Existing tests were updated to use valid hex identifiers.

Hex case normalization

The migration applies lower() inline during conversion without modifying topic_legacy at rest. This means two rows that differ only by hex casing (e.g., g-ABCD vs g-abcd) collapse to the same binary topic. A deduplication step (keep newest by ID) runs before recreating the unique index.

String topic fidelity in delivery payloads

The "topic" string in delivery payloads is now reconstructed from the binary topic via TopicToString(), rather than forwarded verbatim from Envelope.ContentTopic. This means the string will always be lowercase hex. Consumers should treat topic strings as case-insensitive and prefer "topicBytesB64" when available.

Envelope passthrough

The raw Envelope (containing string ContentTopic) is still passed through in SendRequest.Message for payload delivery and idempotency key generation. Only subscription lookups and message classification use the binary topic.

TopicString is stored, not computed

Subscription.TopicString is populated once when reading from DB via TopicToString() and stored on the struct, rather than computed on each access. This avoids repeated hex encoding in delivery hot paths where multiple services read the same subscription.

Test plan

• 70 tests pass (go test -p 1 ./...)
• golangci-lint run — no issues
• go build ./... and ./dev/build — binary compiles
• Topic conversion: 14 tests covering ParseV3Topic, TopicToString, GetMessageTypeFromTopic
• Migration: data conversion test with conforming, non-conforming, and case-duplicate rows
• SQLC: BYTEA round-trip test
• Subscriptions: 9 tests including nil topic guard
• API: 19 tests including string topics, bytes topics, merged/deduped, invalid, empty, SubscribeWithMetadata precedence
• Listener: 4 tests with binary topic conversion and non-convertible topic drop
• Delivery: APNS and FCM payload tests verifying dual topic fields
• Interfaces: JSON marshaling tests for both populated and nil topic cases

🤖 Generated with Claude Code

Note

Migrate subscription topics from text to binary V4 format

• Adds topic_bytes / topics_bytes binary fields to the Subscription, SubscribeRequest, and UnsubscribeRequest proto messages, allowing clients to pass V4 binary topics alongside or instead of string topics.
• Adds DB migration 00004_binary_topics.up.sql that converts subscriptions.topic from TEXT to BYTEA, parsing V3 topic strings into kind-prefixed bytes, deleting non-convertible rows, and deduplicating on (installation_id, topic).
• Updates the Subscriptions interface and all implementations so Subscribe, Unsubscribe, and GetSubscriptions accept *topic.Topic objects instead of strings; SQL queries are updated to use bytea[] parameters throughout.
• Adds topics.ParseV3Topic and topics.TopicToString helpers for converting between string and structured topic representations; removes the old GetMessageType(envelope) function in favour of GetMessageTypeFromTopic(*topic.Topic).
• Updates APNS and FCM delivery to populate the notification topic field from Subscription.Topic rather than envelope.ContentTopic.
• Risk: the up migration deletes rows whose topic cannot be parsed as a valid V3 topic string and collapses duplicates, which is irreversible without the down migration.

^{Macroscope summarized c7d7e68.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR implements a significant migration from string-based to binary topic format, involving a database schema migration that transforms existing data, changes to core subscription interfaces, new API fields, and a new external dependency. These are substantial runtime behavior changes with schema impact. Additionally, there is an unresolved review comment identifying a potential bug in error handling.

^{You can customize Macroscope's approvability policy. Learn more.}

[neekolas]

• ci: add concurrency limits to all workflows #100
• feat: migrate test infrastructure to xnet #98
• feat: add V4 d14n listener with payload format support #96
• refactor: replace local upstream protos with xmtpd imports #97
• feat: migrate to V4 binary topic format #94 👈 (View in Graphite)
• refactor: centralize test utilities and standardize test patterns #99
• chore: upgrade Go from 1.25 to 1.26 #93
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[neekolas]

Merge activity

• Apr 8, 5:03 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Apr 8, 5:05 PM UTC: Graphite rebased this pull request as part of a merge.
• Apr 8, 5:05 PM UTC: @neekolas merged this pull request with Graphite.

[macroscopeapp]

🟢 Low

example-notification-server-go/pkg/xmtp/listener.go

Line 192 in c7d7e68

return err

At line 192, return err returns the last err value from the loop, but this is wrong in two ways: (1) if a deliver call fails and logs an error, execution continues and the stale err is returned at the end, causing startMessageWorkers to log a duplicate error for a failure that was already handled; (2) if the final iteration skips delivery via shouldDeliver (hitting continue), err still holds the error from a previous iteration's failed deliver, so processEnvelope returns a stale error for a message that was otherwise processed successfully.

-	return err
+	return nil

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file pkg/xmtp/listener.go around line 192:

At line 192, `return err` returns the last `err` value from the loop, but this is wrong in two ways: (1) if a `deliver` call fails and logs an error, execution continues and the stale `err` is returned at the end, causing `startMessageWorkers` to log a duplicate error for a failure that was already handled; (2) if the final iteration skips delivery via `shouldDeliver` (hitting `continue`), `err` still holds the error from a *previous* iteration's failed `deliver`, so `processEnvelope` returns a stale error for a message that was otherwise processed successfully.

Evidence trail:
pkg/xmtp/listener.go lines 180-192 (loop with deliver and return err), pkg/xmtp/listener.go lines 127-139 (startMessageWorkers logs returned error). Reviewed at REVIEWED_COMMIT.