Complement TestPartialStateJoin/Can_change_display_name_during_partial_state_join is flakey #15086

Activity

1. DMRobertson

   added

   T-TaskRefactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.

   Z-FlakeTests that give intermittent failuresTests that give intermittent failures

   A-TestingIssues related to testing in complement, synapse, etcIssues related to testing in complement, synapse, etc

   A-ProfilesDisplaynames, avatarsDisplaynames, avatars

   on Feb 16, 2023
2. DMRobertson

   added this to the Remaining faster remote join work milestone on Feb 16, 2023
3. squahtx commented on Feb 17, 2023

   

   squahtx

   on Feb 17, 2023

   Last edited by squahtx

   Contributor

   More actions

   It looks like the federation sender for the complement hostname:port in question was stuck in a retry-backoff from an earlier test. So raising the timeout may not be sufficient to deflake this failure.

   2023-02-16T18:15:17.1268502Z federation_sender1 | 2023-02-16 17:58:24,592 - synapse.federation.sender.transaction_manager - 121 - INFO - federation_transaction_transmission_loop-69 - TX [host.docker.internal:33229] {1676570133846} Sending transaction [1676570133846], (PDUs: 0, EDUs: 1)
   
   2023-02-16T18:15:17.1317260Z federation_sender1 | 2023-02-16 17:58:24,896 - synapse.http.federation.matrix_federation_agent - 363 - INFO - federation_transaction_transmission_loop-69 - Failed to connect to host.docker.internal:33229: Connection was refused by other side: 111: Connection refused.
   2023-02-16T18:15:17.1318122Z federation_sender1 | 2023-02-16 17:58:24,896 - synapse.http.matrixfederationclient - 672 - INFO - federation_transaction_transmission_loop-69 - {PUT-O-73} [host.docker.internal:33229] Request failed: PUT matrix://host.docker.internal:33229/_matrix/federation/v1/send/1676570133846: ConnectionRefusedError('Connection refused')
   
   2023-02-16T18:15:17.1506818Z federation_sender1 | 2023-02-16 17:58:29,579 - synapse.http.federation.matrix_federation_agent - 363 - INFO - federation_transaction_transmission_loop-69 - Failed to connect to host.docker.internal:33229: Connection was refused by other side: 111: Connection refused.
   2023-02-16T18:15:17.1507631Z federation_sender1 | 2023-02-16 17:58:29,579 - synapse.http.matrixfederationclient - 672 - INFO - federation_transaction_transmission_loop-69 - {PUT-O-73} [host.docker.internal:33229] Request failed: PUT matrix://host.docker.internal:33229/_matrix/federation/v1/send/1676570133846: ConnectionRefusedError('Connection refused')
   
   2023-02-16T18:15:17.2006041Z master             | 2023-02-16 17:58:45,894 - synapse.replication.http.membership - 89 - INFO - POST-525 - remote_join: @t45alice:hs1 into room: !0-XCj272ZvqAN6H5W06J:host.docker.internal:33229
   
   2023-02-16T18:15:17.2036181Z event_creator1     | 2023-02-16 17:58:46,435 - synapse.access.http.18013 - 460 - INFO - PUT-155 - ::ffff:127.0.0.1 - 18013 - {@t45alice:hs1} Processed request: 0.210sec/0.002sec (0.008sec, 0.021sec) (0.011sec/0.067sec/22) 2B 200 "PUT /_matrix/client/v3/profile/@t45alice:hs1/displayname HTTP/1.0" "Go-http-client/1.1" [4 dbevts]
   
   2023-02-16T18:15:17.2086530Z federation_sender1 | 2023-02-16 17:58:49,380 - synapse.http.federation.matrix_federation_agent - 363 - INFO - federation_transaction_transmission_loop-69 - Failed to connect to host.docker.internal:33229: Connection was refused by other side: 111: Connection refused.
   2023-02-16T18:15:17.2087337Z federation_sender1 | 2023-02-16 17:58:49,380 - synapse.http.matrixfederationclient - 672 - INFO - federation_transaction_transmission_loop-69 - {PUT-O-73} [host.docker.internal:33229] Request failed: PUT matrix://host.docker.internal:33229/_matrix/federation/v1/send/1676570133846: ConnectionRefusedError('Connection refused')
   

   Note that all three "Failed to connect" log lines are for the same PUT-O-73 transaction from before the failing test case.
4. squahtx

   added

   A-Federated-Joinjoins over federation generally suckjoins over federation generally suck

   Z-Dev-WishlistMakes developers' lives better, but doesn't have direct user impactMakes developers' lives better, but doesn't have direct user impact

   on Feb 17, 2023
5. DMRobertson commented on Feb 17, 2023

   

   DMRobertson

   on Feb 17, 2023

   ContributorAuthor

   More actions

   I wonder if we can get Synapse to clear out its backoff data when a new deployment is made?
6. DMRobertson commented on Feb 20, 2023

   

   DMRobertson

   on Feb 20, 2023

   ContributorAuthor

   More actions

   https://github.com/matrix-org/synapse/actions/runs/4215073441/jobs/7315978888

   time="2023-02-19T07:29:44Z" level=error msg="Failed to fetch key for server" context=missing error="Post "[https://127.0.0.1:33010/_matrix/key/v2/query](https://127.0.0.1:33010/_matrix/key/v2/query/)": context canceled" fetcher=DirectKeyFetcher
   2023/02/19 07:29:44 complement: Transaction '1676791589957': HTTP Code 401. Invalid http request: {Invalid request signature}

   Might be the same problem?
7. squahtx

   changed the title [-]Complement `Can_change_display_name_during_partial_state_join` is flakey[/-] [+]Complement `TestPartialStateJoin/Can_change_display_name_during_partial_state_join` is flakey[/+] on Feb 21, 2023
8. squahtx commented on Mar 29, 2023

   

   squahtx

   on Mar 29, 2023

   Last edited by squahtx

   Contributor

   More actions

   Presumed to be fixed by matrix-org/complement#626.
9. squahtx

   closed this as completedon Mar 29, 2023
10. anoadragon453 commented on May 5, 2023

    

    anoadragon453

    on May 5, 2023

    Member

    More actions

    Unfortunately, I just experienced a flake on this test in https://github.com/matrix-org/synapse/actions/runs/4893678003/jobs/8737133564?pr=15544#step:7:14018 (the PR branch is only 2 commits behind current develop).
11. anoadragon453

    reopened this on May 5, 2023