fix(engine): isolate search store from async load crashes#369
Open
QuinnWilton wants to merge 1 commit into
Open
fix(engine): isolate search store from async load crashes#369QuinnWilton wants to merge 1 commit into
QuinnWilton wants to merge 1 commit into
Conversation
This was referenced Feb 10, 2026
QuinnWilton
force-pushed
the
fix/store-async-load-isolation
branch
from
3e18e7d to
ac24ce1
Compare
Replace Task.async with Task.Supervisor.async_nolink in
State.prepare_backend_async/2 so a crashing index build no longer
kills the Store GenServer.
Add a {:DOWN, ...} handler that logs the error and clears
async_load_ref, allowing the Store to retry on the next
project_compiled event. Demonitor the ref on successful completion
to avoid processing the normal DOWN.
mhanberg
force-pushed
the
fix/store-async-load-isolation
branch
from
ac24ce1 to
291a8b8
Compare
mhanberg
reviewed
Feb 14, 2026
Comment on lines
+490
to
+491
| # Allow time for the task to crash and the DOWN message to be processed. | ||
| Process.sleep(100) |
Member
There was a problem hiding this comment.
I suggest having the function that raises send a messsage to the test process instead, and we can assert_receive on that instead of waiting.
| {:DOWN, ref, :process, _pid, reason}, | ||
| {update_ref, %State{async_load_ref: ref} = state} | ||
| ) do | ||
| Logger.error("Search index async load crashed: #{inspect(reason)}") |
Member
There was a problem hiding this comment.
Suggested change
| Logger.error("Search index async load crashed: #{inspect(reason)}") | |
| Logger.error("Failed to prepare search store backend: #{inspect(reason)}") |
| assert_eventually alive?() | ||
|
|
||
| on_exit(fn -> | ||
| after_each_test(Ets, project) |
Member
There was a problem hiding this comment.
Suggested change
| after_each_test(Ets, project) | |
| destroy_backend(Ets, project) |
Member
There was a problem hiding this comment.
this after each test function can probably be deleted, but we can deal with that in a different pr (not you, just as a note for myself)
doorgan
pushed a commit
that referenced
this pull request
Jun 24, 2026
This is part of a set of 4 PRs that arose out of some static analysis tooling I'm working on: - #369 - #370 - #371 - #372 That means that these aren't crashes or issues that I've observed in practice, however based on my reading of the code, they do represent issues worth addressing. ## Problem: `Beams.apply_to_all/2` spawns its worker process using a bare `spawn/1`, and then blocks in `block_until_done/3` until it receives a `:progress` message from it. If the worker crashes, this message will never arrive, and the caller will block indefinitely. ## Solution Replace `spawn/1` with `spawn_monitor/1`, and add a `receive` clause in `block_until_done/3` to handle the worker unexpectedly terminating before a `:progress` message is sent, so that we're able to immediately raise instead of blocking forever. In the successful case, the monitor is flushed to avoid `:DOWN` messages building up and causing a mailbox leak.
doorgan
requested changes
Jun 24, 2026
doorgan
left a comment
doorgan
left a comment
Contributor
There was a problem hiding this comment.
I'm not sure this PR fixes the issue it claims to fix.
If I'm understanding this correctly, this will prevent the Store from crashing, but will leave it in a broken state where it's alive, but nothing was loaded and nothing can be loaded either. That would be hiding a crash and for a user it would still be broken.
I think this needs more changes to get the Store into a healthy state after recovering from a load crash, or to crash even more loudly if this would render the engine unusable
doorgan
pushed a commit
that referenced
this pull request
Jun 24, 2026
This is part of a set of 4 PRs that arose out of some static analysis tooling I'm working on: - #369 - #370 - #371 - #372 That means that these aren't crashes or issues that I've observed in practice, however based on my reading of the code, they do represent issues worth addressing. ## Problem: This is effectively the same issue as in #370, where the use of a bare `spawn/2` causes the spawning process to hang indefinitely while waiting for a message from the spawned process, and preventing `Credo` diagnostics from being updated. ## Solution Replace `spawn/1` with `spawn_monitor/1`, and add a `receive` clause to handle the spawned process unexpectedly terminating before sending a result. In the successful case, the monitor is flushed to avoid `:DOWN` messages building up and causing a mailbox leak. A 30 second timeout is also added, to prevent buggy or broken `Credo` modules from causing the entire system to hang.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is part of a set of 4 PRs that arose out of some static analysis tooling I'm working on:
That means that these aren't crashes or issues that I've observed in practice, however based on my reading of the code, they do represent issues worth addressing.
Problem:
State.prepare_backend_async/2usesTask.async/1to build the search index. If this task raises, theStorecrashes, and the task silently fails. The store is supervised under a:one_for_onestrategy, and as far as I can tell, no indexed data is permanently lost, however:Storewill be returned:noprocinstead of{:error, :loading}, and the crash will cascadeproject_compiledevents are dispatched between the crash and the restart, these will be missed by theStore. Since these events are never replayed, the event will be lost, and the server will remain unloaded until the next compilationSolution
By using
Task.Supervisor.async_nolink/2to start the search index task under a dedicatedEngine.TaskSupervisor, we can instead handle a crash in the task by logging the error, and clearingasync_load_ref, so that the server is immediately ready to process further requests.During this time, the server remains available, callers are isolated from the crash, no
project_compiledevents are lost while waiting for the restart, and previously loaded indexes remain loaded.In the successful case, the monitor is flushed to avoid
:DOWNmessages building up and causing a mailbox leak.