Port synapse_port_db to async/await by babolivier · Pull Request #6718 · matrix-org/synapse

babolivier · 2020-01-16T16:39:34Z

Also raise an exception if there are pending background updates so we return with a non-0 code in the CI if something wrong happened when running the background updates.

Fixes #6719

So we return with a non-0 code

tadzik · 2020-01-16T21:13:09Z

Running this locally still fails for me. As is:

Preparing sqlite3...
2020-01-16 21:56:47,381 - synapse_port_db - 601 - ERROR - 
Traceback (most recent call last):
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 501, in run
    allow_outdated_version=True,
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 471, in build_db_store
    db_conn, allow_outdated_version=allow_outdated_version
TypeError: check_database() got an unexpected keyword argument 'allow_outdated_version'
Traceback (most recent call last):
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 501, in run
    allow_outdated_version=True,
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 471, in build_db_store
    db_conn, allow_outdated_version=allow_outdated_version
TypeError: check_database() got an unexpected keyword argument 'allow_outdated_version'

Removing that parameter pushes it further, but it still dies:

Preparing sqlite3...
2020-01-16 21:57:45,737 - synapse.storage.prepare_database - 304 - INFO - Upgrading schema to v56
Preparing psycopg2...
2020-01-16 21:57:45,771 - synapse_port_db - 601 - ERROR - 
Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 516, in run
    self.hs_config.get_single_database()
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 471, in build_db_store
    db_conn
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/synapse/storage/engines/postgres.py", line 36, in check_database
    txn.execute("SHOW SERVER_ENCODING")
AttributeError: 'psycopg2.extensions.connection' object has no attribute 'execute'
Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 516, in run
    self.hs_config.get_single_database()
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 471, in build_db_store
    db_conn
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/synapse/storage/engines/postgres.py", line 36, in check_database
    txn.execute("SHOW SERVER_ENCODING")
AttributeError: 'psycopg2.extensions.connection' object has no attribute 'execute'

babolivier · 2020-01-17T11:15:37Z

@tadzik I looked into these logs and the corresponding code, it looks like it's caused by the changes in #6675 which haven't been released yet, hence why the patched script is working fine with a Synapse on the develop branch.

Given my local tests and the CI (which is now fixed, and passing), I'm pretty confident that the script now does the right thing.

So to try this patch, instead of blindly applying it onto your local version of synapse_port_db, there are two options:

Setup a development environment for Synapse and checkout the babolivier/port_db_bg_update_fix branch
Wait for the next release which will ship this fix (which should happen next week, or on the following one I think)

tadzik · 2020-01-17T12:28:26Z

Alright, having checked your branch (I'm at 75d60c2), I'm still getting errors (though different ones now :)). They originate from synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py, which I've modified as follows for debugging's sake:

--- a/synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py
+++ b/synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py
@@ -68,6 +68,8 @@ def run_upgrade(cur, database_engine, config, *args, **kwargs):
                 INNER JOIN room_memberships AS r USING (event_id)
                 WHERE type = 'm.room.member' and state_key like '%' || ?
         """
+    print(f"sql is '{sql}'")
+    print(f"config.server_name is '{config.server_name}'")
     cur.execute(sql, (config.server_name,))
 
     cur.execute(

The crash looks like so:

2020-01-17 13:25:39,547 - synapse.storage.prepare_database - 373 - INFO - Running script 57/local_current_membership.py
sql is '
            INSERT INTO local_current_membership (room_id, user_id, event_id, membership)
                SELECT c.room_id, state_key AS user_id, event_id, r.membership
                FROM current_state_events AS c
                INNER JOIN room_memberships AS r USING (event_id)
                WHERE type = 'm.room.member' and state_key like '%' || ?
        '
config.server_name is 'tadzik.net'
(...)
Traceback (most recent call last):
  File "/home/synapse/synapse/synapse.git/scripts/synapse_port_db", line 516, in run
    self.hs_config.get_single_database()
  File "/home/synapse/synapse/synapse.git/scripts/synapse_port_db", line 473, in build_db_store
    prepare_database(db_conn, engine, config=self.hs_config)
  File "/home/synapse/synapse/synapse.git/synapse/storage/prepare_database.py", line 85, in prepare_database
    data_stores=data_stores,
  File "/home/synapse/synapse/synapse.git/synapse/storage/prepare_database.py", line 376, in _upgrade_existing_database
    module.run_upgrade(cur, database_engine, config=config)
  File "/home/synapse/synapse/synapse.git/synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py", line 73, in run_upgrade
    cur.execute(sql, (config.server_name,))
ValueError: unsupported format character ''' (0x27) at index 349

Some further debugging revealed that the offending character is the leading ' in state_key like '%'. I don't see anything wrong with it myself, but perhaps you do :)

babolivier · 2020-01-17T15:19:26Z

Looks like that was #6727, fixed in #6728. I've merged the latest state of develop into my branch to pull the fix in, so feel free to git pull and try again.

babolivier · 2020-01-17T16:21:21Z

@tadzik just told me it worked and he's now live on PostgreSQL, so it looks like everything works correctly now 🎉

* commit '07124d028': Port synapse_port_db to async/await (#6718)

babolivier added 4 commits January 16, 2020 16:38

Raise an exception if there are pending background updates

e697ca0

So we return with a non-0 code

Changelog

c684227

Port synapse_port_db to async/await

7fd6a4e

Port update_database to async/await

07a33a5

babolivier changed the title ~~Raise an exception if there are pending background updates~~ Port synapse_port_db to async/await Jan 16, 2020

babolivier marked this pull request as ready for review January 16, 2020 17:12

babolivier requested a review from a team January 16, 2020 17:12

babolivier added 5 commits January 16, 2020 17:26

Add version string to mocked homeservers

3fa6e54

Remove unused imports

068d185

Convert overseen bits to async/await

f77d2d5

Fixup logging contexts

a4d4e35

Fix imports

75d60c2

Merge branch 'develop' into babolivier/port_db_bg_update_fix

9bdeae5

babolivier mentioned this pull request Jan 17, 2020

Background await fix #6736

Closed

3 of 4 tasks

erikjohnston reviewed Jan 20, 2020

View reviewed changes

Comment thread scripts/synapse_port_db

Add a way to print an error without raising an exception

8179108

babolivier requested a review from a team January 20, 2020 14:39

erikjohnston approved these changes Jan 21, 2020

View reviewed changes

Comment thread scripts/synapse_port_db

Comment thread scripts/synapse_port_db

Incorporate review

0ff6648

babolivier merged commit 07124d0 into develop Jan 21, 2020

richvdh mentioned this pull request Jan 23, 2020

synapse_port_db: RuntimeWarning: coroutine 'BackgroundUpdater.do_next_background_update' was never awaited #6719

Closed

clokep mentioned this pull request Jul 30, 2020

Convert inlineCallbacks to async/await #7988

Closed

19 of 48 tasks

babolivier pushed a commit that referenced this pull request Sep 1, 2021

Port synapse_port_db to async/await (#6718)

09e9ebd

* commit '07124d028': Port synapse_port_db to async/await (#6718)

babolivier deleted the babolivier/port_db_bg_update_fix branch October 28, 2021 15:54

Navigation Menu

Repository navigation

Port synapse_port_db to async/await - #6718

Port synapse_port_db to async/await#6718
babolivier merged 12 commits into
developfrom
babolivier/port_db_bg_update_fix

babolivier commented Jan 16, 2020 •

edited

Loading

Uh oh!

tadzik commented Jan 16, 2020

Uh oh!

babolivier commented Jan 17, 2020 •

edited

Loading

Uh oh!

tadzik commented Jan 17, 2020

Uh oh!

babolivier commented Jan 17, 2020

Uh oh!

babolivier commented Jan 17, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

babolivier commented Jan 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tadzik commented Jan 16, 2020

Uh oh!

babolivier commented Jan 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tadzik commented Jan 17, 2020

Uh oh!

babolivier commented Jan 17, 2020

Uh oh!

babolivier commented Jan 17, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

babolivier commented Jan 16, 2020 •

edited

Loading

babolivier commented Jan 17, 2020 •

edited

Loading