Repository navigation
Conversation
One failing test per known gap of migration mode (RUNLOOM_MIGRATION=1), each marked xfail(strict=True) so it runs in CI, shows as xfailed, and turns into a hard XPASS failure the moment a fix lands. Each scenario runs in a fresh subprocess with a watchdog and a forced migration (a foreign OS thread wakes the fiber, so it goes through the global run-queue), and skips loudly as NOMIG when no migration was observed, so an unpatched interpreter cannot make it pass. Names are test_<area>_<invariant>: the area (harness, memory, preempt, sched, identity, cost) is the subsystem a fix lands in, the invariant the behaviour that must hold. Reproduced on main at H=4: the three brc merge-queue leaks (finished fibers and 452 MiB of cross-hub bytes retained without a GC), timers starved behind a CPU-bound fiber, sleepers lost under introspection, single-thread fibers sharing a context while M:N is live, a run-queued fiber surviving mn_fini, an offload hub pulling general work, 611 us p99 foreign-thread wake latency, stock RLock / importlib / sqlite3 / sys._current_frames() after a migration, and the per-fiber-tstate costs (0.65 us per parked fiber in gc.collect, 33 KiB RSS, 2.9 us per spawn). Three tests pass and guard what was verified to work.
CI is meant to go red on these until each gap is closed. Each test's docstring now carries the finding its former xfail reason named.
johng
changed the title
run_isolated keeps a 30-line tail of a failing file, and each scenario's multi-line failure message let it show only the last two or three failures. The message is now one line (rc + the key error line) and the full transcript goes to captured stdout, so the tail shows the whole failing set.
On the 3-core macOS CI runner a foreign thread's unbuffered send ran before the fiber had parked in recv; a plain thread cannot block, so the runtime raised and the scenario waited out its watchdog, which turned the harness guard and three real reproducers into watchdog exits. Foreign sends now retry until the receiver is parked. The three cost tests no longer carry a platform number: each scenario runs twice, migration on and off, and the bound is a ratio (2x for gc.collect per parked fiber and spawn, 1.5x for RSS per parked fiber). Measured here: 0.38 vs 0.045 us (8x), 3.4 vs 0.55 us (6x), 33 vs 17 KiB (1.97x).
…livery On 3.13 the per-parked-fiber gc.collect cost is ~0.13 us on Linux, so 5k fibers gave a 0.65 ms signal that a loaded 3-core runner's noise in the empty-collection baseline could swallow (one upstream retry passed the 2x bound). 10k fibers, best of 7. The signal-delivery stress asked for 50 deliveries in 2.5 s; handlers run on the main thread's ~16 ms poll, which a busy runner starves. 20 in 3 s, still an order of magnitude above zero deliveries.
Under migration every object is allocated on the running hub's thread id (alloc-home), so a last decref from another hub is queued by biased refcounting to the FIRST thread state in that id's bucket: the hub's own, created first on its thread. That queue is merged only when its owner runs bytecode, is deleted, or a GC starts, and a hub thread state never runs bytecode -- so every cross-hub drop (a finished fiber's G handle held by a WaitGroup parker, a 1 MiB bytes sent over a channel) survived until the next GC, which untracked objects never trigger. Two pieces. The hub services its own merge queue (_Py_HandlePending on the explicit-merge bit) after every resume and on idle turns. And for the duration of a resume the FIBER's thread state takes the hub's thread id and the head of the hub's bucket (runloom_iframe_brc_adopt), so drops during a long resume land on the fiber and its own eval loop merges them at the next eval-breaker check; on park the hub's state goes back to the head and whatever is still queued on the fiber is drained while it is attached (runloom_iframe_brc_release). Both take the bucket mutexes, so both run attached. test_memory_*: 3 of 3 pass; cross-hub bytes peak RSS 452 MiB -> flat.
Under migration the hub's own thread state is detached for the whole resume, so the ATTACHED-wedge test never fired and sysmon stood preemption down: a CPU-bound fiber held its hub and every timer and fd parked there for as long as it ran (1.5 s spinner -> 1.5 s stall). The hub now publishes the thread state its resume runs on (h->resume_tstate: the fiber's under migration, its own otherwise) from resume_begin to resume_end, and sysmon classifies that. A per-g thread state is freed when its fiber completes, so the read goes through a hazard pointer: sysmon publishes the pointer it is about to read and re-checks it is still the hub's current resume; the deleter (runloom_g_decref) fences and waits while the hazard names its state. hubs() reads the same way, so a CPU-spinning hub is reported attached again. test_preempt_cpu_bound_fiber_does_not_starve_its_hubs_timers passes.
A queued g holds its scheduler ref and the queue ref wake_g took on PARKED->QUEUED; the fini drain dropped one, so a fiber woken just before an early run() exit survived with its PyThreadState.
… sweeper The pop had already taken the g off the sleep heap and won the enqueue claim; when introspection held the g SWEEPING (a stack dump, an idle madvise) the PARKED->RUNNING CAS failed and the g was dropped: on no heap, on no queue, never resumed (fibers(stacks=True) under load lost 90 of 128 sleepers). Mirror wake_g: SWEEPING -> SWEEPING_WOKEN, and runloom_mn_sweep_claim_release enqueues it.
The global run-queue pull popped the head for any hub, so an idle offload hub took general fibers and stranded them behind its next blocking call (the one general-work path left onto an offload hub). An offload puller now takes nothing from the queue and its idle scan ignores it; its own submits still wake it through hub_submit. This tree has no per-fiber pin, so the reverse gap is unchanged: a woken offload fiber on the global run-queue is run by a general hub.
runloom_per_g_tstate_mode is process-wide, so while an M:N run was live a single-thread scheduler on another OS thread skipped the pystate snap and the private-context copy: its fibers shared ContextVars and lost exc_info across a park. Gate both on runloom_mn_current_sched().
With the idle backoff off (the default) no hub ever registers for wakep, so a foreign-thread wake -- the one path that cannot hand its g to a hub -- woke nobody, and the g waited for the next 1 ms idle pump of whichever hub pulled it: p99 380-600 us at a sub-2 ms cadence against ~30 us on the per-hub scheduler. The push now kicks ONE hub the way hub_submit kicks a sub_list target: the pinned hub, else the first general hub announcing an idle-condvar wait, else the g's owner hub (it has the g pending, so it sits in a timed pump wait). The hub re-checks the run-queue counters after its idle announce, with SEQ_CST fences on both sides, so the kick and the announce cannot miss each other. test_sched_foreign_thread_wake_reaches_a_shallow_idle_hub_promptly: p99 510 us -> passes the 200 us bound.
tstate->thread_id was the SPAWNER's thread for the fiber's whole life, so sys._current_frames(), sys._current_exceptions() and faulthandler never showed the running fiber under the id it reports through threading.get_ident(), and several fibers collapsed onto one key. While a fiber runs its state now carries the hub's thread id (and native id); while parked it carries its own address, which is no thread's, so a parked fiber cannot shadow the running one. _current_frames skips frameless states and lets the oldest duplicate win: that is the hub's own state, frameless while a fiber runs, so the fiber is what shows.
johng
changed the title
…t xfails Three OS-thread-identity checks (stock RLock and sqlite3 compare the OS thread id in C; importlib's _ModuleLock keys on _thread.get_ident()) and three costs of one PyThreadState per fiber (gc.collect per parked fiber, the 16 KiB datastack chunk, PyThreadState_New per spawn) stay open on purpose. Each is a strict xfail under TODO_MIGRATION_FAIL, so it still runs, shows as xfailed, and turns into a hard XPASS failure the moment its gap is closed. The suite is green; grep TODO_MIGRATION_FAIL for the list.
…tion off A fixed 200 us bound was too tight for the 3-core macOS runner (252 us after the fix, down from 5.7 ms) and a looser one would not catch the 1 ms idle pump on Linux (495 us before the fix). Like the cost tests, the scenario now runs with migration on and off and asserts p99_on < 2 * p99_off + 100 us.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. One failing test per known gap of cross-hub migration mode (
RUNLOOM_MIGRATION=1), with a fix for each gap the runtime can close. The six that remain are strict xfails underTODO_MIGRATION_FAIL, so CI is green and any of them passing later is a hard XPASS failure that forces the marker off. The findings come from a review of migration mode on thestackweavefork (johng#23); this is the same test module as johng#26 ported to this tree's names, and every gap reproduces here onmain.Harness
Each scenario runs in a fresh subprocess with
RUNLOOM_MIGRATION=1, a watchdog (a wedge is a clean exit, not a hung pytest), and a forced migration: a foreign OS thread wakes the fiber so it goes through the global run-queue (the send retries until the fiber is parked, since a plain thread cannot block). If no migration is observed (an unpatched interpreter gates the mode off) the scenario printsNOMIGand the test skips loudly instead of passing.Names are
test_<area>_<invariant>. The area (harness,memory,preempt,sched,identity,cost) is the subsystem a fix lands in, so-k memoryselects one gap family. The invariant is the behaviour that must hold, so the name reads as a plain guard once the gap is closed. The three cost tests run their scenario twice, migration on and off, and bound the ratio, so they mean the same thing on a laptop and on a 3-core CI runner.What each one does today
Observed on the patched 3.14.4t build at H=4 on this tree's
main:test_memory_cross_hub_g_handle_drop_frees_the_fiber_without_gctest_memory_waitgroup_fanout_frees_finished_fibers_without_gctest_memory_cross_hub_bytes_drop_frees_them_without_gctest_preempt_cpu_bound_fiber_does_not_starve_its_hubs_timerstest_sched_timer_pop_survives_a_concurrent_introspection_sweeptest_sched_single_thread_fibers_stay_isolated_while_mn_is_livetest_sched_mn_fini_frees_a_fiber_left_on_the_global_runqsubmittedfiber survivesrun()test_sched_offload_hub_never_pulls_unpinned_general_worktest_sched_foreign_thread_wake_reaches_a_shallow_idle_hub_promptlytest_identity_stock_rlock_releases_after_a_migrationRuntimeError: cannot release un-acquired locktest_identity_sqlite_connection_works_after_a_migrationtest_identity_module_import_lock_releases_after_a_migrationtest_identity_current_frames_lists_the_running_fiber_under_get_identtest_cost_gc_collect_per_parked_fiber_within_2x_of_migration_offtest_cost_parked_fiber_rss_within_1_5x_of_migration_offtest_cost_spawn_and_complete_within_2x_of_migration_offThe memory rows share one root cause: a cross-hub last decref is queued by biased refcounting to the allocating hub's own thread state, which never runs bytecode, so nothing merges the queue until a GC. The fix direction is for the hub loop to service its own merge queue (
_Py_HandlePendingwhen the explicit-merge bit is set) after it re-attaches its thread state.Passing, kept as guards for what was verified to work: the harness premise (a foreign-thread wake migrates),
PyGILState_Ensurecallbacks running on the fiber's own thread state, and signal delivery into a migrating io-sleeper (about 140 deliveries per run). That last one drives the sleep-heap race the review confirmed by reading; it did not reproduce.Fixes (2026-09-26)
Ten of the sixteen gaps are closed on this branch, one commit each, and their tests now pass. The six that remain are strict xfails under
TODO_MIGRATION_FAIL(grep it for the list).test_memory_*pass; cross-hub bytes peak RSS 452 MiB to flattest_preempt_*passeswake_gdoestest_sched_timer_pop_*passestest_sched_single_thread_*passesmn_finileaks a run-queued fibertest_sched_mn_fini_*passestest_sched_offload_hub_*passestest_sched_foreign_thread_wake_*passes, p99 510 us to under 200 ussys._current_frames()test_identity_current_frames_*passesOpen under
TODO_MIGRATION_FAIL, and why:test_identity_stock_rlock_*,test_identity_sqlite_connection_*: stock_thread.RLockand sqlite3'scheck_same_threadcompare the OS thread id in C. Nothing in the runtime can satisfy them once a fiber moves; the answers areG.pin(fork) ormonkey.patch()for the RLock. Needs a decision: keep red, document as a limitation, or makerun()pin fibers that hold such objects (not detectable).test_identity_module_import_lock_*: importlib's_ModuleLockkeys on_thread.get_ident()at Python level. A fiber-awareget_ident(what gevent does) fixes it, but changesthreading.get_ident()semantics for every user, so it belongs inmonkey.patch()and is a design call.test_cost_*(3): onePyThreadStateper fiber. GC visits every parked state, RSS carries each state's 16 KiB datastack chunk, spawn paysPyThreadState_New. A thread-state pool that reuses a finished fiber's state for the next spawn would cut spawn and RSS churn; GC and the chunk are CPython's.Regression check: the full isolated suite (237 files, -j8) passes apart from
test_monkey_leakandtest_greenlet_interop, both of which fail identically on the pre-fix build on this machine and pass in CI. One unexplained event: in one of two full-suite samples the signal-delivery stress hit its watchdog under load; it did not recur in the second sample nor in 18 heavily loaded targeted runs.Differences from the fork's copy
This tree has no per-fiber pin (
G.pin, johng#24) and nomn_current_hub(), so the pinned-fiber guard is left out, the cross-hub bytes test counts cross-hub drops by the allocating thread id instead of pinning producer and consumer, and the offload test learns the offload hub's OS thread from anoffload_fiber.