Hierarchy pagination is limited to 50 items per space #11134

Activity

1. aofn

   changed the title [-]API `_matrix/client/unstable/org.matrix.msc2946/rooms({roomId}/hierarchy` pagination broken[/-] [+]API `_matrix/client/unstable/org.matrix.msc2946/rooms/{roomId}/hierarchy` pagination broken[/+] on Oct 20, 2021
2. squahtx

   added

   T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.Bugs, crashes, hangs, security vulnerabilities, or other reported issues.

   A-SpacesHierarchical organization of roomsHierarchical organization of rooms

   X-Needs-Discussion

   on Oct 20, 2021
3. squahtx commented on Oct 20, 2021

   

   squahtx

   on Oct 20, 2021

   Last edited by squahtx

   Contributor

   More actions

   Are you always seeing the same incorrect number of rooms returned, or does it differ between pagination attempts? And is it the same subset of rooms returned every time?

   What's the structure of the space with 135 children? ie. how many rooms and subspaces are there in each subspace?
4. clokep commented on Oct 20, 2021

   

   clokep

   on Oct 20, 2021

   Member

   More actions

   Is this a public space? If so, can you provide the room ID?
5. squahtx

   added

   X-Needs-InfoThis issue is blocked awaiting information from the reporterThis issue is blocked awaiting information from the reporter

   and removed

   X-Needs-Discussion

   on Oct 20, 2021
6. aofn commented on Oct 20, 2021

   

   aofn

   on Oct 20, 2021

   Last edited by aofn

   Author

   More actions

   Are you always seeing the same incorrect number of rooms returned, or does it differ between pagination attempts?

   Yes, I always get the same incorrect number of rooms returned for a specific space (but the number varies from one space to another).

   And is it the same subset of rooms returned every time?

   Yes, always the same rooms.

   What's the structure of the space with 135 children? ie. how many rooms and subspaces are there in each subspace?

   The room structure is as follows:

   parent space
     | children space 1
     | children space 2
     | children space 3
     ...
     | children space 134
     | children space 135
   

   no further sub-spaces

   Is this a public space? If so, can you provide the room ID?

   I've created a public dummy space with 120 space children here: !iUCIUjePntDkDGSnjb:matrix.org
   In this case the second API call with the pagination token only returns one room.

   edit: just to verify, GET https://matrix.org/_matrix/client/r0/rooms/!iUCIUjePntDkDGSnjb:matrix.org/state
   has 120 events of the typem.space.child
7. squahtx

   removed

   X-Needs-InfoThis issue is blocked awaiting information from the reporterThis issue is blocked awaiting information from the reporter

   on Oct 21, 2021
8. squahtx commented on Oct 21, 2021

   

   squahtx

   on Oct 21, 2021

   Contributor

   More actions

   I suspect what's happening here is that only the first 50 children of each subspace are being returned.
   @marcel-klasse does this line up with what you're seeing?
9. aofn commented on Oct 21, 2021

   

   aofn

   on Oct 21, 2021

   Author

   More actions

   that's possible. I have just added 5 children to the first subspace like this:

   parent space
     | children space 1
        | child 1
        | child 2
        | child 3
        | child 4
        | child 5
     | children space 2
     | children space 3
     ...
     | children space 119
     | children space 120
   

   which is now returning 4 rooms after the second call.
10. zv1n commented on Nov 22, 2021

    

    zv1n

    on Nov 22, 2021

    More actions

    I'm seeing this exact issue when trying to migrate a list of rooms from another service over to a matrix server space.

    The issue appears to be that the 50 room limit is hard coded on line 52 in handlers/room_summary.py with the request cap limiting logic appearing on line 653 in handlers/room_summary.py.

    The limiting logic appears to come in from PR Federation API for Space summary (#9652) where the max_children check against MAX_ROOMS_PER_SPACE is first introduced into _summarize_local_room.

    I've tested this on a local test VM. When I set this variable (MAX_ROOMS_PER_SPACE) to 100, it works as expected (I have 90 rooms and got 90 rooms).

    The easiest solution would seem to be expose this as a configuration variable so individual servers can override the limit as needed, but I figured I'd drop my research in for the synapse team to consider.

    Thanks!
11. clokep

    changed the title [-]API `_matrix/client/unstable/org.matrix.msc2946/rooms/{roomId}/hierarchy` pagination broken[/-] [+]Hierarchy pagination is limited to 50 items per space[/+] on Nov 23, 2021
12. clokep commented on Nov 23, 2021

    

    clokep

    on Nov 23, 2021

    Member

    More actions

    I've reworded the issue since it isn't really broken, but working as designed. Part of the problem with this code right now is that it is trying to support both the legacy /spaces API and the newer /hierarchy API so it gets a bit messy. We could likely up the limit of 50 higher, but it likely makes more sense to ensure we can handle arbitrary limits. I would need to look deeper into the code to see how easy that is.
13. clokep

    self-assigned this

    on Jan 6, 2022
14. clokep

    closed this as completedin #11695on Jan 8, 2022