Repository navigation
MSC4140: finalised delayed events - #19038
AndrewFerr wants to merge 62 commits into
Conversation
- Store sent/cancelled/failed delayed events, i.e. finalised delayed events, and support looking them up to inspect whether a delayed event was sent or not. Set limits on how many finalised events to store. - Support looking up delayed events by ID - Return 200 when retrying the same action (send/cancel) on an already-finalised delayed event, or 409 for a conflicting action - Limit how many delayed events a user may have scheduled at a time
| @@ -0,0 +1 @@ | |||
| Add more support for MSC4140, namely the ability to inspect sent, cancelled, or failed delayed events, aka "finalised" delayed events. | |||
There was a problem hiding this comment.
Are there Complement changes to go along with this?
I see some TestDelayedEvents Complement tests that are failing: https://github.com/element-hq/synapse/actions/runs/18411628889/job/52465384686?pr=19038
There was a problem hiding this comment.
Are there Complement changes to go along with this?
No, not yet. I'll add some soon.
I see some
TestDelayedEventsComplement tests that are failing
That's an unrelated failure, which has been flaky for a frustratingly long time. Maybe now is a good time to try to tackle it again.
There was a problem hiding this comment.
It doesn't seem flaky. It's failed all 3 times for both SQLite and Postgres. And it's only TestDelayedEvents
There was a problem hiding this comment.
Complement tests are tracked by matrix-org/complement#929
| -- See the GNU Affero General Public License for more details: | ||
| -- <https://www.gnu.org/licenses/agpl-3.0.html>. | ||
|
|
||
| CREATE TABLE finalised_delayed_events ( |
There was a problem hiding this comment.
What the difference between delayed_events.is_processed and the state in finalised_delayed_events?
It looks like finalised_delayed_events holds more info but I'm not immediately seeing the difference.
There was a problem hiding this comment.
is_processed is for delayed events that have just timed out and are in the process of being sent / persisted to their target room's DAG. Only once a delayed event is successfully sent does it get finalised. (Prior to this PR, it would instead get deleted from the DB.)
The purpose of tracking is_processed is to handle the edge case of the server going down after a delayed event times out, but before it gets sent. Upon server restart, the sending of any is_processed delayed events will be retried.
I'm admittedly not a big fan of this, but I couldn't find a way to make an atomic action out of a delayed event timing out & being sent. I also didn't want to fiddle with that in this PR, even if finalised events now have some redundancy with is_processed events.
| for user_localpart in self.db_pool.simple_select_onecol_txn( | ||
| txn, | ||
| "finalised_delayed_events", | ||
| keyvalues={}, | ||
| retcol="DISTINCT(user_localpart)", | ||
| ): | ||
| self._prune_excess_finalised_delayed_events_for_user( | ||
| txn, user_localpart, retention_limit | ||
| ) |
There was a problem hiding this comment.
This is a N+1 query problem.
Can we do this in batches in one big query?
There was a problem hiding this comment.
Not sure, because each prune is a 2-step process of:
- find how many finalised delayed events the target user has
- delete enough of that user's FDEs to stay within the per-user retention limit
I'm not sure of a better way to do it that would involve only a single query.
What should help is the fact this pruning is always done in a transaction, together with whatever other queries need to happen before/after it.
There was a problem hiding this comment.
@barodeur made this into a single query with the help of window functions: #20229 (comment)
The query has been moved around by later commits, but the core concept remains.
Instead of storing details on finalised delayed events in a new, standalone table that must be joined on the base table of delayed events, add columns to the base table to track those details.
ea6b3ca to
048508c
Compare
This satisfies the check-schema-delta script. The index is present for speeding up ORDER BY on finalised_ts.
This should also fix the portdb script
Using DatabasePool.execute always calls fetchall on the transaction, which fails in Postgres if there are no results to fetch. So, use DatabasePool.runInteraction for UPDATE queries that are expected to sometimes make no updates.
The order doesn't really matter, but having two changes share the same ID/order feels inappropriate
Do this to keep typechecking for non-None delayed event details when a result is to be returned. Also skip looking up the next send time on re-send, as recent changes have the caller ignore it in that case.
Keep the recent bugfix where re-cancelling a delayed event would disarm the timer for the next delayed event, but so do in a way that doesn't require looking up the next send time.
For consistency with other docstrings
The per-user retention DELETE ranks finalised rows with `ROW_NUMBER() OVER (PARTITION BY user_localpart ORDER BY finalised_ts DESC)`, which no index matched, so every 5-minute prune sorted all finalised rows. Add a partial background index `delayed_events_finalised_rank` on `(user_localpart, finalised_ts DESC, delay_id ASC) WHERE finalised_ts IS NOT NULL`, registered from the 94/11 delta next to `delayed_events_finalised_ts`, so the window is computed from an index scan instead of a sort. Break ties on `delay_id ASC` so the surviving row is deterministic when rows share a `finalised_ts` (as `cancel_delayed_state_events` produces), and match the outer DELETE on `(user_localpart, delay_id)` so it uses the primary key rather than depending on the `delayed_events_idx` background update.
Will move these to another PR
- Keep only the configs that control enablement of MSC4140 in server.py, and add a short comment explaining that - Tweak the comments preceding the finalised retention configs to better serve as documentation for them - Move the tests for these configs to test_experimental.py
When a batch of timed out delayed events includes state events, simply send those state events in order with all other events to be sent, rather than sending only the first state event per matching state key & doing so after sending message events. This is safe now that delayed state event intererence has been dropped (see element-hq#19495).
When sending a batch of delayed events at their scheduled send time, allow concurrent sending of events that target different rooms.
When breaking ties on which finalised delayed events to prune, prefer to prune those that had been scheduled earlier (i.e. ones with the lowest `delayed_since_ts`, i.e. `send_ts - delay`), followed by those with an earlier intended send time.
Instead of pruning finalised delayed events on a 5 minute timer, prune them as soon as they exceed retention policies. Also make user deactivation finalise cancelled delayed events instead of deleting them, unless the deactivation requests to erase user data.
When trying to send/cancel/restart a delayed event that is in the midst of being sent, return a different error than 404, which would improperly tell clients that the delayed event in question doesn't exist. This is a stopgap measure until there is a way to properly handle such requests in this scenario.
|
After far too long, this PR is finally ready for another round of review. Many thanks to @barodeur for getting it back on track. This pulls in changes from the following PRs: |
AndrewFerr
changed the title
MSC4140: finalised delayed events
Dev notes
Delayed events initially introduced in #17326
Pull Request Checklist
EventStoretoEventWorkerStore.".code blocks.