Description
After upgrading from 1.84.0 to 1.85.0 worker processes are unable to contact the main process.
Downgrading to 1.84.0 with the same configuration resolves the connection problems.
The configuration is using a shared instance_map block with a main section.
Could be that there is something wrong with the existing configuration and that it worked by chance.
Looking at the changelog for 1.85.0, pull request #15578 seems relevant.
Steps to reproduce
Upgrade from 1.84.0 to 1.85.0 using the attached configuraiton.
Homeserver
private server
Synapse Version
1.85.0
Installation Method
Docker (matrixdotorg/synapse)
Database
PostgreSQL, single server. Not Not restored from backup.
Workers
Multiple workers
Platform
Master process and workers are running as containers in a k8s cluster.
Configuration
Relevant parts of shared homeserver.yaml
pid_file: /var/run/homeserver.pid
log_config: "/config/log.config"
tls_private_key_path: "/etc/ssl/cluster-certificate/tls.key"
tls_certificate_path: "/etc/ssl/cluster-certificate/tls.crt"
listeners:
- port: 8008
tls: true
bind_address: '0.0.0.0'
type: http
x_forwarded: true
resources:
- names: [client]
compress: false
- names: [federation]
compress: false
- port: 9093
bind_address: '0.0.0.0'
tls: true
type: http
resources:
- names: [replication]
instance_map:
main:
host: synapse.default.svc
port: 9093
tls: true
synapse_worker_0:
host: synapse-worker-0.synapse-worker-headless.default.svc
port: 9093
tls: false
synapse_worker_1:
host: synapse-worker-1.synapse-worker-headless.default.svc
port: 9093
tls: false
stream_writers:
events:
- synapse_worker_0
- synapse_worker_1
worker_replication_secret: "secret"
redis:
enabled: true
host: "redis.default.svc"
port: 6379
password: "redis-password"
worker.yaml
worker_name: synapse_worker_0
worker_app: synapse.app.generic_worker
worker_log_config: "/config/log.config"
worker_listeners:
- port: 8008
tls: true
bind_addresses: ['0.0.0.0']
type: http
x_forwarded: true
resources:
- names: [client]
compress: false
- names: [federation]
compress: false
- port: 9093
bind_address: '0.0.0.0'
tls: false
type: http
resources:
- names: [replication]
Relevant log output
Following is observed on the worker nodes
2023-06-07 05:26:49,967 - synapse.http.client - 933 - INFO - GET-53- Error sending request to POST synapse-replication://master/_synapse/replication/presence_set_state/<redacted>: AttributeError 'str' object has no attribute 'decode'
2023-06-07 05:26:50,395 - synapse.federation.federation_server - 1488 - INFO - PUT-55- Failed to handle edu 'm.presence': SynapseError('502: Failed to talk to master process')Anything else that would be useful to know?
No response
Description
After upgrading from 1.84.0 to 1.85.0 worker processes are unable to contact the main process.
Downgrading to 1.84.0 with the same configuration resolves the connection problems.
The configuration is using a shared
instance_mapblock with amainsection.Could be that there is something wrong with the existing configuration and that it worked by chance.
Looking at the changelog for 1.85.0, pull request #15578 seems relevant.
Steps to reproduce
Upgrade from 1.84.0 to 1.85.0 using the attached configuraiton.
Homeserver
private server
Synapse Version
1.85.0
Installation Method
Docker (matrixdotorg/synapse)
Database
PostgreSQL, single server. Not Not restored from backup.
Workers
Multiple workers
Platform
Master process and workers are running as containers in a k8s cluster.
Configuration
Relevant parts of shared homeserver.yaml
worker.yaml
Relevant log output
Anything else that would be useful to know?
No response