Skip to content
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.

History purge failed with: ValueError: max() arg is an empty sequence #10669

Closed
Closed
History purge failed with: ValueError: max() arg is an empty sequence#10669
Labels
S-MinorBlocks non-critical functionality, workarounds exist.T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.good first issueGood for newcomers

Description

@Kokokokoka
Kokokokoka
opened on Aug 21, 2021

Description

On history purge, this happened:

2021-08-20 21:25:22,304 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626834204073 => stream_or
dering 6232155)
2021-08-20 23:17:30,415 - synapse.handlers.pagination - 286 - ERROR - _purge_history-5673 - [purge] failed
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/pagination.py", line 279, in _purge_history
    await self.storage.purge_events.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/purge_events.py", line 54, in purge_history
    state_groups = await self.stores.main.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 49, in purge_history
    return await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 686, in runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 791, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
    raise exception.with_traceback(traceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 786, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 554, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 105, in _purge_history_txn
    max_depth = max(row[1] for row in rows)
ValueError: max() arg is an empty sequence
2021-08-20 23:17:30,443 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626841050432 => stream_or
dering 6232211)
2021-08-20 23:17:30,453 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626841050443 => stream_or
dering 6232211)
2021-08-20 23:19:03,094 - synapse.handlers.pagination - 286 - ERROR - _purge_history-5691 - [purge] failed
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/pagination.py", line 279, in _purge_history
    await self.storage.purge_events.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/purge_events.py", line 54, in purge_history
    state_groups = await self.stores.main.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 49, in purge_history
    return await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 686, in runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 791, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
    raise exception.with_traceback(traceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 786, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 554, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 105, in _purge_history_txn
    max_depth = max(row[1] for row in rows)
ValueError: max() arg is an empty sequence
2021-08-20 23:19:15,824 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626841155788 => stream_or
dering 6232211)

Then there's no connection with element (crash?)

Version information

Synapse: 1.40,
on Debian 11,
using matrix repo.
This is a migration from 1.0 version

Activity

  1. erikjohnston
    added
    good first issueGood for newcomers
    S-MinorBlocks non-critical functionality, workarounds exist.
    T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.
    on Aug 23, 2021

  2. erikjohnston commented on Aug 23, 2021

    @erikjohnston
    erikjohnston
    on Aug 23, 2021
    Member

    I think the fix here is to handle the case of the DB query returning no rows.

  3. anoadragon453 commented on Aug 26, 2021

    @anoadragon453
    anoadragon453
    on Aug 26, 2021
    Member

    What caused the purge out of interest? Do you have the experimental room retention feature enabled, or was the room purged manually?

  4. Kokokokoka commented on Aug 26, 2021

    @Kokokokoka
    Kokokokoka
    on Aug 26, 2021
    ContributorAuthor

    I answered in a PR reply, sorry, I'll copy it here:
    I would love to find out the cause of this issue. The problem is, that we migrated from 1.0->1.41. Rooms, messages, files got cleared by some mystique scripts before (maybe this is the cause of the bug, i'm trying to find this out in the db).
    Now there is message retention policy, it helps a lot. I don't know if there's a policy for empty rooms/rooms with only deleted accounts which weren't accessed for x. I had to manually implement local files|thumbnails cleaner (which checks what elems have last access ts > x, and then purge the elems which aren't in the db.

    For instance, how could you have a room graph that doesn't have any leaves?

    Can I see this in the db somehow?
    (For now I did:
    empty rooms:
    select count(*) from rooms where not exists(select from room_memberships where room_memberships.room_id = rooms.room_id); to find empty rooms (26)
    and:
    rooms that become empty if deactivated|ppl with activity > x ago are substracted:

    
select count(*) from rooms where room_id not in(select room_id from room_memberships where
user_id not in (select name from users where name in (select user_id from (select  
user_id, max(timestamp) ts from user_daily_visits group by user_id) x where ts < 
1627215050807) or deactivated = 1));
count
-------
 1421
(1 row))

    rooms with 1 active:

    select count(*) from (select room_id, count(user_id) cn from room_memberships where 
user_id not in (select name from users where name in (select user_id from (select  
user_id, max(timestamp) ts from user_daily_visits group by user_id) x where ts < 
1627215050807) or deactivated = 1) group by room_id)x where cn =1;
count
796
(1 row)

    So, I don't really know whether this room was purged manually, all I know that there was some script that managed local_media+should have managed messages and we had syn v1.0

  5. richvdh commented on Sep 23, 2021

    @richvdh
    richvdh
    on Sep 23, 2021
    Member

    There are a few known reasons why we could end up with a room without any forward-extremities: see #8094 which is another symptom of such a situation. Some examples include:

    • There was a background update introduced in Synapse 1.10 (Backgroud update to clean out rooms from current state #6802) which removes the forward-extremities for any room where there are no active members.
    • Ill-advised database surgery, including running various scripts that can be found on the internet.
  6. richvdh
    added a commit that references this issue on Sep 24, 2021

    In `_purge_history_txn`, ensure that txn.fetchall has elements before…

    e704cc2

  7. callahad commented on Oct 5, 2021

    @callahad
    callahad
    on Oct 5, 2021
    Contributor

    Now that #10690 is merged, can we close this?

  8. Kokokokoka commented on Oct 6, 2021

    @Kokokokoka
    Kokokokoka
    on Oct 6, 2021
    ContributorAuthor

    I think so, if there's no objections, i'll gladly do this.

  9. Kokokokoka
    closed this as completedon Oct 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    S-MinorBlocks non-critical functionality, workarounds exist.T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.good first issueGood for newcomers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions