Skip to content

internal.log_lines: preserve microsecond timestamp order across writer batches - #3550

Open
jshearer wants to merge 2 commits into
jshearer/discovers-graphql-codexfrom
jshearer/discovers-log-ordering-codex
Open

jshearer wants to merge 2 commits into
jshearer/discovers-graphql-codexfrom
jshearer/discovers-log-ordering-codex

Conversation

@jshearer

@jshearer jshearer commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Description:

The log writer gives every line in a batch the same timestamp, so timestamp-based pagination can skip the rest of a batch. Assign each line a timestamp at PostgreSQL's microsecond precision, at least one microsecond beyond the previous line and no earlier than the current clock reading. Retain the previous timestamp across batches and backward clock adjustments.

Deployment:

This PR is stacked on #3545. Deploy this writer fix before or with the discovery API; when landing the stack in order, deploy after both PRs have landed. #3553 has no dependency on this stack.

Validation:

The real-writer pagination regression passes in the API, agent, and models suite and in the combined run with current master and #3553. This commit builds with the offline SQL cache.

@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from 4f6286c to ed69f22 Compare September 29, 2026 19:09
@jshearer
jshearer added this pull request to stack #3555 September 29, 2026 19:28
@jshearer jshearer changed the title logs: preserve microsecond timestamp order across writer batches internal.log_lines: preserve microsecond timestamp order across writer batches Sep 29, 2026
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch 2 times, most recently from ed69f22 to 2f9af2c Compare September 30, 2026 19:26
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch 2 times, most recently from 43e9704 to 9eb7824 Compare September 30, 2026 21:03
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from 9eb7824 to 5c0254b Compare October 1, 2026 15:13
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from 5c0254b to 3b580e8 Compare October 1, 2026 17:48
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from 3b580e8 to 350a7fc Compare October 1, 2026 19:36
@jshearer
jshearer marked this pull request as ready for review October 1, 2026 19:52
@strix-security

strix-security Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Strix Security Review

No security issues found.

Review summary

Reviewed all three files in PR #3550. The production change in crates/control-plane-api/src/logs.rs assigns each internal.log_lines row a monotonic, microsecond-precision logged_at timestamp derived solely from the server clock and the previous batch's timestamp, and binds it as the fourth parameter of the existing insert ... unnest($1,$2,$3,$4) statement. The query remains fully parameterized across token, stream, log_line, and logged_at, so no attacker-controlled data reaches a dangerous sink. The new next_logged_at helper operates only on server-generated clock values, and its panic path is unreachable for realistic timestamps. The remaining two files are test-only: a #[cfg(test)]-gated integration test refactored to exercise the real writer path, and its corresponding snapshot updated to redact clock-derived values. No injection, authentication/authorization, secret-handling, cryptography, deserialization, or other security-relevant change was identified.

Updated for b3dbc25.


Reviewed by Strix
Re-run review 路 Configure security review settings

@jshearer jshearer self-assigned this Oct 1, 2026
@jshearer
jshearer requested a review from GregorShear October 1, 2026 19:52
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch 2 times, most recently from 4e96ba0 to ad03a2b Compare October 5, 2026 15:36
@jshearer
jshearer requested a review from bbartman October 5, 2026 16:27
bbartman
bbartman previously approved these changes Oct 6, 2026

@bbartman bbartman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from ad03a2b to bedd659 Compare October 6, 2026 14:19
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from bedd659 to a4e9a75 Compare October 6, 2026 14:38
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from a4e9a75 to e957aa0 Compare October 6, 2026 14:47
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch 2 times, most recently from 9277957 to 2762d5a Compare October 6, 2026 18:28
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from 2762d5a to 8651884 Compare October 6, 2026 18:43
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch 2 times, most recently from 713ac42 to f62d598 Compare October 6, 2026 19:20
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch 2 times, most recently from 228b081 to fc6a768 Compare October 6, 2026 21:04
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from fc6a768 to ef9ce8a Compare October 6, 2026 21:14
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from ef9ce8a to 8781cd8 Compare October 6, 2026 22:29
Add createDiscover using an owned draft's capture or a readable live capture.
Validate authorization, connector readiness, and placement before atomically
inserting the discover row and scheduling discovery.

Preserve serialized definitions and publication preconditions. Cover staged and
live capture selection, byte-preserved endpoint configuration, the update_only
policy, rejections, data plane selection, capability checks, and the
discovery-to-publication workflow.
Assign each line a timestamp at PostgreSQL's microsecond precision, advancing
past the writer's previous timestamp even when its clock moves backward.
Retain that timestamp across batches so timestamp cursors can resume within
and between batches without skipping lines from this writer.

Exercise the real writer through the discover log connection, including page
boundaries within a batch, across batches, and late writes. Keep a focused unit
test for clock recession. Ordering remains local to one writer's lifetime.
@jshearer
jshearer force-pushed the jshearer/discovers-log-ordering-codex branch from 8781cd8 to b3dbc25 Compare October 7, 2026 01:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants