Repository navigation
decouple harvester app from harvesting #6027
Description
User Story
In order to process jobs for catalog when harvester app is down, datagov wants to decouple harvesting from the harvester admin app
Acceptance Criteria
[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]
- GIVEN the harvester admin app is coupled with harvesting
WHEN the harvesting app and harvesting admin app are separated
THEN the harvester app doesn't impact the harvesting process
[AND optionally another verifiable outcome]
Background
[Any helpful contextual notes or links to artifacts/evidence, if needed]
Security Considerations (required)
[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]
Sketch
[Notes or a checklist reflecting our understanding of the selected approach]
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
FuhuXia commentedon Sep 22, 2026
We should not use instance 0 of the web app to call LoadManager.start() to start harvest jobs. Doing so creates a dependency on the web app being up and running.
Instead, I’m going to invoke LoadManager.start() only from apps launched via
cf run-task. This decouples harvest processing from the web app, allowing harvest jobs to continue even when the web app is scaled down to zero instances.Further improvement:
GitHub cron is not reliable. If we start to miss too many schedules, we can explore cloud.gov's Run periodic scheduled tasks.
FuhuXia commentedon Sep 23, 2026
harvest is running on all spaces.