Move on-the-side database processing into update_events.py

[taldcroft]

Description

Previously, task_schedule_events.cfg shelled out to mkdir/cp/mv around each kadi_update_events cron run to process a scratch copy of events3.db3, then unconditionally moved that copy back over the production file — even when nothing had actually changed. Since the events database only changes roughly once every 8 hours (when new telemetry arrives), the production file was being needlessly touched on every 10-minute tick.

Changes

• Add a --process-dir option to update_events.py. When set (absolute, or relative to --data-root), the production events3.db3 is copied into that directory and processed there instead of in place.
• After processing, the copy is shallow compared against the production file (checking size and date); the production file is only replaced (via atomic os.replace) if it actually changed. If unchanged, production is left untouched.
• The copy step itself is skipped when the process-side copy already exists and matches production, avoiding a redundant file copy on every run when nothing changed upstream.
• If --update-dir is not provided, behavior is unchanged: the database is updated in place in --data-root.
• Extracted the event-model update loop into a standalone update_events() function so main() can run the compare/move logic after it regardless of early returns (e.g. "no new MAUDE telemetry").
• Simplified task_schedule_events.cfg to a single kadi_update_events --data-root=... --update-dir=update_events --maude call, removing the separate mkdir/cp/mv exec lines.

Test plan

• Verified via a standalone smoke test (stubbing the model-update loop) that: in-place mode works unchanged, an unchanged process copy is not moved back, a changed copy is atomically moved back, absolute --update-dir paths work, and the copy step is skipped/re-triggered correctly across repeated runs.
• ruff check passes on update_events.py.
• Manual validation against a real events3.db3 with --maude in a staging-like environment before deploying the updated cron config.

Interface impacts

Testing

Unit tests

• Mac

(ska3) ➜  kadi git:(update-events-on-local-copy) git rev-parse --short HEAD
72bce8e2
(ska3) ➜  kadi git:(update-events-on-local-copy) pytest --lf               
=========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.13.11, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/aldcroft/git
configfile: pytest.ini
plugins: anyio-4.12.1, timeout-2.4.0
collected 318 items                                                                                                                                                       
run-last-failure: 471 known failures not in selected tests

kadi/commands/tests/test_commands.py .......................s..............................                                                                         [ 16%]
kadi/commands/tests/test_states.py ...............................................x..........................                                                       [ 40%]
kadi/commands/tests/test_validate.py ......................                                                                                                         [ 47%]
kadi/commands/tests/test_commands_v2.py ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss                                    [ 74%]
kadi/commands/tests/test_filter_events.py ..                                                                                                                        [ 75%]
kadi/commands/tests/test_observations.py ...........................................                                                                                [ 88%]
kadi/tests/test_events.py ..........                                                                                                                                [ 92%]
kadi/tests/test_events_connection.py ..                                                                                                                             [ 92%]
kadi/tests/test_occweb.py .......................                                                                                                                   [100%]

========================================================= 228 passed, 89 skipped, 1 xfailed in 117.85s (0:01:57) ==========================================================

Independent check of unit tests by [REVIEWER NAME]

• [PLATFORM]:

(ska3-latest) flame:kadi jean$ git rev-parse HEAD
72bce8e2db0f69cdd446374996068105ee1d5772
(ska3-latest) flame:kadi jean$ pytest
========================================= test session starts ==========================================
platform darwin -- Python 3.13.11, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/jean/git
configfile: pytest.ini
plugins: anyio-4.12.1, timeout-2.4.0, dash-4.4.1
collected 318 items                                                                                                                                  

kadi/commands/tests/test_commands.py .......................s..............................                                                    [ 16%]
kadi/commands/tests/test_commands_v2.py ssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 34%]
ssssssssssssssssssssssssssssssss                                                                                                               [ 44%]
kadi/commands/tests/test_filter_events.py ..                                                                                                   [ 45%]
kadi/commands/tests/test_observations.py ...........................................                                                           [ 58%]
kadi/commands/tests/test_states.py ...............................................x............. [ 77%]
.............                                                                                                                                  [ 82%]
kadi/commands/tests/test_validate.py ......................                                                                                    [ 88%]
kadi/tests/test_events.py ..........                                                                                                           [ 92%]
kadi/tests/test_events_connection.py ..                                                                                                        [ 92%]
kadi/tests/test_occweb.py .......................  

Functional tests

Local mac testing by hand

On my mac, I copied the events3.db3 from Ska data. It happened to be 6 days old because I hadn't synced since then. I then ran the following commands from the top of the git repo:

python -m kadi.scripts.update_events --data-root=. --process-dir=process_events --maude --stop=2026-08-08
python -m kadi.scripts.update_events --data-root=. --process-dir=process_events --maude --stop=2026-08-08
python -m kadi.scripts.update_events --data-root=. --process-dir=process_events --maude --stop=2026-08-08
python -m kadi.scripts.update_events --data-root=. --process-dir=process_events --maude --stop=2026-08-08T06:00:00
python -m kadi.scripts.update_events --data-root=. --process-dir=process_events --maude --stop=2026-08-08T08:00:00
python -m kadi.scripts.update_events --data-root=. --process-dir=process_events --maude --stop=2026-08-08T08:00:00.001
python -m kadi.scripts.update_events --data-root=. --process-dir=process_events --maude --stop=2026-08-08T08:00:00.001

After each command I examine the files in process_events, confirming that either the process_events/events3.db3 was there and unchanged, or that it had been moved to ./events3.db3. The logging outputs were as expected.

Soak test on linux (kady)

I set up a dedicated dev ska and it is now running every 10 minutes:

5,15,25,35,45,55 * * * * /export/tom/kadi-pr-389/bin/skare task_schedule3.pl -config kadi/task_schedule_events.cfg

As of Aug 18, the log shows that this is working as expected. The only small surprise is that sometimes there are two updates (10 minutes apart) after one comm. This is presumably due to the updates occuring during MAUDE ingest so that both times MAUDE reports having new telemetry. I think this is benign and we do want to ensure that kadi events are fully caught up with MAUDE after each comm.

(base) kady$ grep "Events database changed" kadi_events.log 
2026-08-13 14:06:28,942 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-13 14:16:27,868 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-13 23:06:25,197 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-13 23:16:28,076 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-14 06:46:26,203 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-14 06:56:23,065 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-14 15:06:27,600 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-14 15:16:24,881 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-14 23:36:21,273 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-14 23:46:23,061 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-15 07:16:25,093 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-16 00:16:36,813 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-16 05:26:23,755 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-16 10:36:24,356 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-17 03:46:26,785 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3
2026-08-17 03:56:26,341 main: Events database changed, moving /export/tom/kadi-pr-389/data/kadi/process_events/events3.db3 -> /export/tom/kadi-pr-389/data/kadi/events3.db3

Most of the processing jobs end like this:

2026-08-18 06:16:01,896 main: Events database unchanged, not updating production copy /export/tom/kadi-pr-389/data/kadi/events3.db3

[jeanconn]

Docstring should include a note indicating that --process-dir needs to be on the same disk mount as the flight KADI dir.