Skip to content

feat(plugins): reconcile missed shot uploads - #645

Open
ODevStudio wants to merge 4 commits into
mainfrom
odev/issue-603-shot-reconciliation
Open

feat(plugins): reconcile missed shot uploads#645
ODevStudio wants to merge 4 commits into
mainfrom
odev/issue-603-shot-reconciliation

Conversation

@ODevStudio

@ODevStudio ODevStudio commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

What changed, and why?

  • Capture the DE1 serial, model, firmware version, and flow calibration when a
    shot starts, explicitly clearing stale identity if capture fails, so later
    uploads never use the wrong machine.
  • Reconcile a bounded, paginated backlog while the machine is idle or sleeping.
    Durable success and rejection annotations let restarts resume without
    repeatedly submitting completed or permanently invalid records.
  • Retry transport, server, rate-limit, timeout, and authorization failures on a
    later pass. Pause as soon as brewing starts or automatic upload is disabled.
  • Upgrade existing 0.2.0 installs to 0.2.1 and prioritize newly stored shots
    ahead of the remaining historical backlog.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore / infra
  • Plugin (DYE2 or bundled skin)

Scope (select all touched areas)

  • BLE transport / device comms
  • REST API / handlers
  • WebSocket API
  • Machine state / shot logic
  • Scale / weight / flow
  • Profiles / beans / grinders / workflows
  • WebUI skins
  • Plugins / JS runtime
  • UI / Flutter widgets
  • Storage / Drift database
  • CI / build / infra
  • Docs / specs

Linked Issues

Fixes #603

Issue #603 is ready for agent work. Production-backend validation with a real machine identity remains recommended before merge.

Root Cause (if bug fix)

N/A - this adds bounded reconciliation for uploads missed while the bundled
uploader was unavailable.

Regression Test Plan (if bug fix or refactor)

N/A - feature coverage locks in capture-time identity, candidate filtering,
pagination, cancellation before every retry, durable rejection, and transient retry in
test/plugins/shot_upload_reconciliation_test.dart and related shot tests.

Documentation Obligations (required)

  • API spec updated: assets/api/rest_v1.yml or assets/api/websocket_v1.yml (if REST/WebSocket changed)
  • API docs updated: doc/Api.md (if user-facing endpoint changed)
  • Plugin docs updated: doc/Plugins.md (if events/API changed)
  • Skin docs updated: doc/Skins.md (if skin behavior changed)
  • Profile docs updated: doc/Profiles.md (if profile handling changed)
  • Device docs updated: doc/DeviceManagement.md (if device flows changed)
  • N/A - no docs affected

Security Impact (required)

  • New or changed REST endpoints? Yes; existing shot workflow JSON gains
    optional capture-time machine provenance.
  • New or changed WebSocket topics? No.
  • New or changed network calls? Yes; the opt-in bundled uploader can replay a
    bounded backlog through its existing authenticated endpoint.
  • BLE/USB surface changed? No.
  • File system access changed? No; existing local shot storage is reused.
  • Plugin sandbox boundary changed? Yes; the bundled uploader declares the
    existing events.machine permission needed to pause reconciliation safely.
  • Risk and mitigation: Upload stays opt-in, uses capture-time identity, pauses
    outside idle/sleep states, is bounded and paginated, persists terminal
    outcomes, and reuses the backend's idempotent machine/shot key.

User-Visible Changes

  • Enabling automatic upload now catches up eligible local shots without waiting
    for another shot.
  • Legacy records without trustworthy machine identity remain local rather than
    being attributed to the currently connected machine.

Verification

Local gates (run before pushing)

  • dart format lib test - no remaining candidate changes
  • flutter analyze - clean
  • flutter test - 3,218 passed, 1 skipped; 1 existing Windows CRLF assertion
  • ./scripts/fetch_dye2_plugin.sh - not rerun for this local draft

Manual verification (if applicable)

  • OS / platform tested: Windows test host and Samsung SM-X210 tablet on Android 15.
  • Simulated devices? (simulate=1): Yes; used for the Android tablet UI smoke check.
  • Real hardware? (DE1/Bengle/scale): No; the physical Android tablet was used without coffee hardware.
  • What you personally verified and how: Shot capture, filtering, pagination,
    retry classification, persistence, and pause behavior through focused tests.
  • Edge cases checked: Lost responses, permanent rejection, authorization and
    transport failures, brewing mid-batch, disabled upload, and missing identity.
  • What you did not verify: Production backend or real-machine upload.

Evidence

  • Test output (failing before + passing after)
  • Log snippets
  • Screenshot / recording (UI changes)
  • curl / websocat output (API changes)

Detailed evidence:

  • Regression-first coverage exercises capture-time identity, candidate
    filtering, exact pagination progress, active and unavailable machine states,
    disabling reconciliation mid-batch or during retry backoff, brewing during retry backoff, permanent rejection, and retry after transient or authorization failures.
  • flutter test --no-pub test/models/workflow_context_test.dart test/controllers/de1_state_manager_shot_state_test.dart test/plugins/bundled_plugin_permissions_test.dart test/plugins/shot_upload_plugin_test.dart test/plugins/shot_upload_reconciliation_test.dart (66 passed)
  • flutter analyze --no-pub (no issues)
  • flutter test --no-pub with the package-matched QuickJS DLL on PATH
    (3,186 passed, 1 skipped)
  • git diff --cached --check before commit (clean)
  • Review follow-up commit 7039af8 re-checks unload, opt-out, and machine state before every POST attempt.
  • Focused upload suites passed 14 tests, including cancellation during retry backoff.
  • Merge commit 62c7a43 incorporates current main, including reference commit 0ea6034. The replacement reconciliation remains authoritative while captured mock serials are still excluded.
  • Latest focused provenance, plugin, controller, workflow, permission, retry, and cancellation suites passed 44 tests.
  • The latest Windows full suite reached 3,218 passed and 1 skipped; its only failure was the existing CRLF-sensitive AsyncAPI assertion.
  • Samsung SM-X210 Android 15 simulated-mode smoke: the Decent shot upload v0.2.1 card and Settings dialog render normally; only AutoUpload and LengthThreshold are present, and the old Upload existing shot history toggle is absent. The dialog was cancelled without saving.
  • The backend's documented canonical key is <machine serial>/<shot id>, so a
    replay after a lost response targets the same stored shot rather than minting
    a duplicate. This was confirmed from the maintainer discussion on PR Add shot-upload plugin + enable plugin Decent-proxy writes #565.
  • Attempted flutter run --no-pub -d windows --dart-define=simulate=1. The
    local build stopped before launch because Flutter selected CMake 3.20 while
    the Firebase SDK requires CMake 3.22 or newer. No production-backend or real
    machine smoke test was run.

Compatibility & Migration

  • Backward compatible? Yes; machine provenance fields are optional and
    existing shot records remain valid.
  • Config / env changes needed? No.
  • Database migration needed? No; annotations and existing JSON storage are
    reused.
  • Exact steps: None.

Risks & Mitigations

  • Risk: Reconciliation could duplicate uploads or compete with a live shot.
    • Mitigation: Use the backend's canonical <machine serial>/<shot id> key,
      bounded batches, durable outcomes, and immediate machine-state checks.
  • Risk: Old records could be assigned to the wrong DE1.
    • Mitigation: Upload only records with capture-time identity; leave ambiguous
      legacy records local.

Contributor Responsibility

AI-assisted development is allowed. The submitter remains responsible for the submitted work.

  • I have reviewed and understand all changes in this PR and take responsibility for their correctness, security, behavior, licensing, and provenance, including any AI-assisted or AI-generated work.

@ODevStudio

Copy link
Copy Markdown
Collaborator Author

P1 — a failed machine snapshot can persist stale machine identity. In lib/src/controllers/de1_state_manager.dart around 602–617, the code reads the currently connected DE1 and replaces baseWorkflow.machine only when that read succeeds. On failure it persists baseWorkflow unchanged. But Workflow.machine can now contain serialNumber loaded through Workflow.fromJson, and /api/v1/workflow accepts that structure into the current workflow. The reconciler then treats any workflow.machine.serialNumber as trustworthy capture-time provenance.
That means if the fresh machine read fails while the active workflow contains identity from a previously loaded workflow/shot, this new shot can inherit the old serial and later be uploaded under the wrong machine—the exact failure #603 is intended to prevent. I would capture an immutable machine snapshot when the shot sequencer starts, or at minimum explicitly clear provenance when fresh identity cannot be established. Note that the current Workflow.copyWith(machine: null) cannot clear the field because it uses machine ?? this.machine. Add a regression test with a stale serial in baseWorkflow and a failing connectedDe1() read; the resulting shot must have no trusted serial and must not reconcile.
P1 — disabling automatic upload does not stop already-scheduled POST retries. reconcile() checks state.autoUpload, unloaded, and machine safety between records, but once uploadShot() enters postShot(), the three-attempt retry loop never checks any of those conditions.
Reproduction: the first backend POST returns 503; before the retry timer fires, dispatch settingsUpdated with AutoUpload: false. The retry loop will still issue the second—and potentially third—POST, so a later retry can upload the shot after the user has disabled uploads. The same problem applies when stateUpdate changes to espresso: additional retry attempts can start while brewing even though the PR says reconciliation pauses when brewing starts. postShot() should receive/check a continuation predicate before every new POST and after each retry delay. Add tests that inject a transient failure, disable auto-upload or enter espresso before the next timer, and assert only the first request occurred.

@tadelv

tadelv commented Aug 20, 2026

Copy link
Copy Markdown
Member

@ODevStudio please rebase on main and verify there is still anything useful to pull out from this PR.
ref commit: 0ea6034

@ODevStudio ODevStudio closed this Aug 21, 2026
@ODevStudio ODevStudio reopened this Aug 22, 2026
@ODevStudio
ODevStudio marked this pull request as ready for review August 22, 2026 14:27
…econciliation

# Conflicts:
#	assets/plugins/shot-upload.reaplugin/manifest.json
#	assets/plugins/shot-upload.reaplugin/plugin.js
#	test/plugins/shot_upload_plugin_test.dart
@ODevStudio
ODevStudio requested a review from tadelv August 22, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make shot-upload plugin eventually reconcile the full local shot library

2 participants