`KeyError: ('m.room.power_levels', '')` when joining some rooms

[nexy7574]

Description

Some room joins seem to fail with KeyError: ('m.room.power_levels', '') almost immediately after send_join returns, presenting a 500 Internal Server Error to the user, and ultimately preventing the join.
Sometimes retrying the join a few more times succeeds, but increasingly commonly (like in the room in the log snippet) it does not.

Steps to reproduce

• join a remote room for the first time

The error is too vague for me to be able to provide any meaningful reproduction steps. All I did in this case was attempted to join #element-x-ios:matrix.org as I would any other room.

Homeserver

starstruck.systems

Synapse Version

1.151.0rc1

Installation Method

pip (from PyPI)

Database

postgres, mono, clean

Workers

Multiple workers

Platform

Ubuntu LXC, master process and worker processes controlled by systemd, same-metal postgres in another lxc, redis on the same lxc hosting synapse

Configuration

Many experimental features enabled, presence disabled, synapse-http-antispam module enabled - full config at https://starstruck.systems/~star/synapse.yaml

Relevant log output

2026-04-10 18:52:00,459 INFO synapse.http.matrixfederationclient (POST-149674): {GET-O-18866} [matrix.org] Completed request: 200 OK in 0.11 secs, got 3088 bytes - GET matrix-federation://matrix.org/_matrix/federation/v1/query/directory?room_alias=%23element-x-ios%3Amatrix.org
2026-04-10 18:52:00,620 INFO synapse.http.matrixfederationclient (POST-149674): {GET-O-18867} [matrix.org] Completed request: 200 OK in 0.16 secs, got 482 bytes - GET matrix-federation://matrix.org/_matrix/federation/v1/make_join/%21nqXEcZbBVWpiHLmQhVANyKwuRD3d672nXjd1iEKPuTU/%40nex%3Astarstruck.systems?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=7&ver=8&ver=9&ver=10&ver=11&ver=12&ver=org.matrix.msc3757.10&ver=org.matrix.msc3757.11&ver=org.matrix.hydra.11
2026-04-10 18:52:01,413 INFO synapse.http.matrixfederationclient (POST-149674): {PUT-O-18868} [matrix.org] Completed request: 200 OK in 0.79 secs, got 1382050 bytes - PUT matrix-federation://matrix.org/_matrix/federation/v2/send_join/%21nqXEcZbBVWpiHLmQhVANyKwuRD3d672nXjd1iEKPuTU/%24z5m1ksllJoTTzPw73rFDX3JMO9hKy1vsPRjP-Xa3Ezc
2026-04-10 18:52:01,413 INFO synapse.federation.federation_client (POST-149674): Processing from send_join 1860 events
2026-04-10 18:52:01,830 INFO synapse.crypto.keyring (PerspectivesKeyFetcher-72): Requesting keys <...>
2026-04-10 18:51:30,223 INFO synapse.handlers.federation_event (POST-149621): Persisting 753 remaining outliers: ['$iCgYS2KvQ03DJQtygE5uUdbeGvMPV954nO_zhL1nR6Q', '$iCt7sazLa9DSjy341dhUkX9JlV8p6M6-XOfTNdAzn3U', '$iDw578dOuZsG1qmGnOKo7CG6N1bD85Tz1IPQPmZ9a50', '$86fslg5C3fnhbVdv26Rw_JcivdfrpJUTwDQzLBkrrCQ', '$iJJQee_0dFewEva3dSWuMTvwpd4FtUZaBWysL64s7Po', ...]
2026-04-10 18:51:30,516 ERROR synapse.http.server (POST-149621): Failed handle request via 'JoinRoomAliasServlet': <XForwardedForRequest at 0x70e49015f3e0 method='POST' uri='/_matrix/client/v3/join/%23element-x-ios:matrix.org' clientproto='HTTP/1.1' site='http'>
Traceback (most recent call last):
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/http/server.py", line 335, in _async_render_wrapper
    callback_return = await self._async_render(request)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/http/server.py", line 576, in _async_render
    callback_return = await raw_callback_return
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/rest/client/room.py", line 615, in on_POST
    return await self._do(request, requester, room_identifier, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/rest/client/room.py", line 596, in _do
    await self.room_member_handler.update_membership(
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/room_member.py", line 670, in update_membership
    result = await self.update_membership_locked(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/room_member.py", line 1111, in update_membership_locked
    remote_join_response = await self._remote_join(
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/room_member.py", line 1970, in _remote_join
    event_id, stream_id = await self.federation_handler.do_invite_join(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/federation.py", line 716, in do_invite_join
    await self._federation_event_handler.process_remote_join(
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/federation_event.py", line 570, in process_remote_join
    await self._auth_and_persist_outliers(
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/federation_event.py", line 1825, in _auth_and_persist_outliers
    await self.persist_events_and_notify(
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/handlers/federation_event.py", line 2331, in persist_events_and_notify
    ) = await self._storage_controllers.persistence.persist_events(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 435, in persist_events
    ret_vals = await yieldable_gather_results(enqueue, partitioned.items())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/util/async_helpers.py", line 347, in yieldable_gather_results
    raise dfe.subFailure.value from None
  File "/opt/synapse/.venv/lib/python3.12/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 430, in enqueue
    return await self._event_persist_queue.add_to_queue(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 251, in add_to_queue
    res = await make_deferred_yieldable(end_item.deferred.observe())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/twisted/internet/defer.py", line 1187, in __iter__
    yield self
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 294, in handle_queue_loop
    ret = await self._per_item_callback(room_id, item.task)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 376, in _process_event_persist_queue_task
    return await self._persist_event_batch(room_id, task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/controllers/persist_events.py", line 661, in _persist_event_batch
    await self.persist_events_store._persist_events_and_state_updates(
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/databases/main/events.py", line 462, in _persist_events_and_state_updates
    redact_allowed = await self._can_sender_redact(event)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/databases/main/events.py", line 502, in _can_sender_redact
    pl_id = state[(EventTypes.PowerLevels, "")]
            ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/synapse/.venv/lib/python3.12/site-packages/synapse/storage/databases/main/state.py", line 984, in __getitem__
    return super().__getitem__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: ('m.room.power_levels', '')
2026-04-10 18:51:30,518 INFO synapse.access.http.http (POST-149621): 80.3.155.46 - http - {@nex:starstruck.systems} Processed request: 14.069sec/0.000sec ru=(0.318sec, 0.007sec) db=(0.002sec/0.019sec/15) 67B 500 "POST /_matrix/client/v3/join/%23element-x-ios:matrix.org HTTP/1.1" "gomuks/v26.03+dev.ce0c1d61 mautrix-go/v0.26.4+dev.261f1770 go/1.26.1" [1 dbevts]

Anything else that would be useful to know?

No response

[nexy7574]

Had to make the _can_sender_redact call infallible, defaulting to false (which is probably incorrect behaviour, but at least it isn't a crash), when called by _persist_events_and_state_updates:

synapse/synapse/storage/databases/main/events.py

Lines 461 to 467 in 62523d8

	# check that sender can redact
	redact_allowed = await self._can_sender_redact(event)
	# Signal that this user's past events in this room
	# should be redacted by adding an entry to
	# `room_ban_redactions`.
	if redact_allowed:

--- a/synapse/storage/databases/main/events.py
+++ b/synapse/storage/databases/main/events.py
@@ -459,7 +459,10 @@ class PersistEventsStore:
                     continue
 
                 # check that sender can redact
-                redact_allowed = await self._can_sender_redact(event)
+                try:
+                    redact_allowed = await self._can_sender_redact(event)
+                except KeyError:
+                    redact_allowed = False
 
                 # Signal that this user's past events in this room
                 # should be redacted by adding an entry to

I was then able to join the room successfully.