Skip to content

fix(runtime): reconcile workflow timeouts fail closed #1102

Description

@doublewhy

Gap Claim

Workflow timeout reconciliation catches malformed persisted timestamps as if no timeout occurred, and compares the timeout against the wrong clock value. Corrupt state can therefore remain active indefinitely or be reconciled inconsistently.

Existing Surface Audit

Reviewed control_plane_timeouts.py, persisted operation timestamps, timeout state transitions, restart reconciliation, and the timeout fixes currently bundled into PR #1086.

Scope

OpenRAE runtime reliability only. Requirements: RUN-317 and RUN-318.

Chosen Architecture

Parse persisted timestamps through one strict boundary, reject malformed or future-inconsistent values with a stable fail-closed diagnostic, and compare monotonic elapsed duration against the configured workflow timeout without confusing wall-clock timestamps with durations.

Verification Plan

Cover malformed timestamps, timezone offsets, future timestamps, exact boundaries, elapsed-over-limit transitions, restart reconciliation, and stable public diagnostics. Keep unrelated PR #1086 changes outside this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions