feat: add V4 d14n listener with payload format support

[neekolas]

Summary

Add a V4 listener that connects to the XMTP d14n network via NotificationApi.SubscribeAllEnvelopes, receiving OriginatorEnvelope messages. Operators select --listener-type=v4 at startup.

• V4 listener processes OriginatorEnvelope payloads (group messages, welcome messages, welcome pointers)
• Converts V4 envelopes to V3 format for backward-compatible push delivery
• Adds payloadFormat field to APNS/FCM/HTTP delivery payloads
• Installations can register a preferred payload format (v3 or v4)
• Extracts shared delivery dispatch, HMAC sender filtering, and worker pool utilities
• Custom MarshalJSON on SendRequest preserves backward-compatible HTTP delivery JSON shape
• Database migration adds payload_format column to installations

Design: docs/plans/2026-04-03-d14n-listener-design.md

Test plan

• go test -p 1 ./... passes (unit tests including V4 listener, delivery, API)
• Existing V3 listener behavior unchanged
• HTTP delivery JSON format backward-compatible (nested message.content_topic shape preserved)
• APNS/FCM payloads include payloadFormat field

🤖 Generated with Claude Code

Note

Add V4 notification listener with payload format support and binary topic storage

• Introduces V4Listener in pkg/xmtp/v4_listener.go that subscribes to XMTP v4 envelopes via a new NotificationApiClient, converts them to V3-compatible payloads, and dispatches push notifications through the shared deliveryDispatcher.
• Adds PayloadFormat enum (V3/V4) to the notification proto, installation registration, and subscription APIs; APNs and FCM delivery payloads now include a payload_format field.
• Migrates the subscriptions topic column from TEXT to BYTEA (migration 00004) and adds a payload_format column to installations (migration 00005); all subscription queries updated accordingly.
• Extends the API server to accept binary topics (topic_bytes) alongside legacy string topics, deduplicate them, and validate payload format against the configured listener type.
• Adds a --listener-type option (v3/v4, default v3) to select the active listener at startup.
• Risk: the binary topic migration drops rows with non-conforming topic strings and changes deduplication key semantics for idempotency (timestamp now included in buildIdempotencyKey).

^{Macroscope summarized 07882a8.}

[macroscopeapp]

Approvability

Verdict: Needs human review

Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

^{You can customize Macroscope's approvability policy. Learn more.}

[macroscopeapp]

🟢 Low interfaces/interfaces.go:88

When listenerType is an empty string, ValidateForListener allows PayloadFormatV4 to pass validation because the condition listenerType == ListenerTypeV3 evaluates to false. However, an empty listenerType is treated as V3 elsewhere in the codebase, so this validation mismatch permits V4 format registrations that are incompatible with the actual V3 listener behavior. Consider explicitly rejecting empty listenerType in NormalizePayloadFormat or treating empty string as V3 in ValidateForListener to match the listener initialization logic.

 func (p PayloadFormat) ValidateForListener(listenerType ListenerType) error {
 	normalized := NormalizePayloadFormat(p)
-	if listenerType == ListenerTypeV3 && normalized == PayloadFormatV4 {
+	if (listenerType == ListenerTypeV3 || listenerType == "") && normalized == PayloadFormatV4 {
 		return fmt.Errorf("payload format %q is not supported by listener type %q", normalized.String(), listenerType)
 	}
 	return nil
 }

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file pkg/interfaces/interfaces.go around lines 88-94:

When `listenerType` is an empty string, `ValidateForListener` allows `PayloadFormatV4` to pass validation because the condition `listenerType == ListenerTypeV3` evaluates to false. However, an empty `listenerType` is treated as V3 elsewhere in the codebase, so this validation mismatch permits V4 format registrations that are incompatible with the actual V3 listener behavior. Consider explicitly rejecting empty `listenerType` in `NormalizePayloadFormat` or treating empty string as V3 in `ValidateForListener` to match the listener initialization logic.

Evidence trail:
pkg/interfaces/interfaces.go:74-78 (ListenerType definition: `ListenerTypeV3 ListenerType = "v3"`), pkg/interfaces/interfaces.go:88-93 (ValidateForListener implementation), cmd/server/main.go:93-96 (switch statement with default case treating non-"v4" as V3), cmd/server/main.go:106 (listenerType passed directly to API server without normalization)

[neekolas]

• ci: add concurrency limits to all workflows #100
• feat: migrate test infrastructure to xnet #98
• feat: add V4 d14n listener with payload format support #96 👈 (View in Graphite)
• refactor: replace local upstream protos with xmtpd imports #97
• feat: migrate to V4 binary topic format #94
• refactor: centralize test utilities and standardize test patterns #99
• chore: upgrade Go from 1.25 to 1.26 #93
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[neekolas]

Merge activity

• Apr 8, 5:03 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Apr 8, 5:08 PM UTC: Graphite rebased this pull request as part of a merge.
• Apr 8, 5:09 PM UTC: @neekolas merged this pull request with Graphite.

[macroscopeapp]

🟢 Low xmtp/v3_listener.go:234

buildSendRequests hardcodes PayloadFormatV3 at line 234, ignoring Installation.PayloadFormat. Installations registered with PayloadFormatV4 will receive messages in the wrong format. Consider using installation.PayloadFormat to match the V4 listener's dispatch pattern.

-			PayloadFormat:    interfaces.PayloadFormatV3,
+			PayloadFormat:    installation.PayloadFormat,

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file pkg/xmtp/v3_listener.go around line 234:

`buildSendRequests` hardcodes `PayloadFormatV3` at line 234, ignoring `Installation.PayloadFormat`. Installations registered with `PayloadFormatV4` will receive messages in the wrong format. Consider using `installation.PayloadFormat` to match the V4 listener's dispatch pattern.

Evidence trail:
- pkg/xmtp/v3_listener.go lines 219-244 (REVIEWED_COMMIT): `buildSendRequests` function hardcodes `PayloadFormat: interfaces.PayloadFormatV3` at line 234
- pkg/interfaces/interfaces.go lines 106-110: `Installation` struct has `PayloadFormat PayloadFormat` field
- pkg/xmtp/v4_listener.go lines 196-206: V4 listener properly switches on `inst.PayloadFormat` to dispatch to either `buildV4SendRequest` or `buildV3SendRequest`

[macroscopeapp]

🟢 Low xmtp/v3_listener.go:197

processEnvelope returns err at line 197, which holds the last result from the delivery loop. If shouldDeliver skips a subscription after a delivery failure, the stale non-nil err is returned. If a delivery fails but a later delivery succeeds, err is overwritten to nil, swallowing the earlier error. The V4 equivalent processOriginatorEnvelope returns nil unconditionally. Consider returning nil at line 197 to match V4 behavior and avoid incorrect error propagation.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file pkg/xmtp/v3_listener.go around line 197:

`processEnvelope` returns `err` at line 197, which holds the last result from the delivery loop. If `shouldDeliver` skips a subscription after a delivery failure, the stale non-nil `err` is returned. If a delivery fails but a later delivery succeeds, `err` is overwritten to `nil`, swallowing the earlier error. The V4 equivalent `processOriginatorEnvelope` returns `nil` unconditionally. Consider returning `nil` at line 197 to match V4 behavior and avoid incorrect error propagation.

Evidence trail:
pkg/xmtp/v3_listener.go lines 186-197 (REVIEWED_COMMIT): Shows the delivery loop with `if err = l.dispatcher.deliver(request); err != nil` and `return err` at line 197.
pkg/xmtp/v4_listener.go lines 158-230 (REVIEWED_COMMIT): Shows equivalent V4 function with same loop pattern but `return nil` at line 230 instead of `return err`.

[macroscopeapp]

🟢 Low topics/topics.go:14

strings.TrimPrefix returns the string unchanged when the prefix is absent, so ParseV3Topic silently parses topic strings that lack the required V3_PREFIX (e.g., "g-abc123/proto" without the leading /xmtp/mls/1/). Consider checking that the original topicStr actually has the prefix before trimming, returning an error if it does not.

+func ParseV3Topic(topicStr string) (*topic.Topic, error) {
+	if !strings.HasPrefix(topicStr, V3_PREFIX) {
+		return nil, fmt.Errorf("invalid V3 topic: missing %s prefix", V3_PREFIX)
+	}
 	topicStr = strings.TrimPrefix(topicStr, V3_PREFIX)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file pkg/topics/topics.go around lines 14-19:

`strings.TrimPrefix` returns the string unchanged when the prefix is absent, so `ParseV3Topic` silently parses topic strings that lack the required `V3_PREFIX` (e.g., `"g-abc123/proto"` without the leading `/xmtp/mls/1/`). Consider checking that the original `topicStr` actually has the prefix before trimming, returning an error if it does not.

Evidence trail:
pkg/topics/topics.go lines 12-39 at REVIEWED_COMMIT: Shows V3_PREFIX constant as "/xmtp/mls/1/" on line 12, ParseV3Topic function starting at line 14, and TrimPrefix call on line 15 without any check that the prefix was actually present. Go standard library documentation confirms strings.TrimPrefix returns the original string unchanged if the prefix is not present.