Move update_client_ips to the background worker #10425

Activity

1. anoadragon453

   added

   T-EnhancementNew features, changes in functionality, improvements in performance, or user-facing enhancements.New features, changes in functionality, improvements in performance, or user-facing enhancements.

   on Jul 19, 2021
2. anoadragon453 commented on Jul 19, 2021

   

   anoadragon453

   on Jul 19, 2021

   Last edited by anoadragon453

   MemberAuthor

   More actions

   I think all that needs to happen is wrapping the looping_call in a conditional for run_background_tasks_on:

   synapse/synapse/storage/databases/main/client_ips.py

   Lines 435 to 452 in c0df6ba

   	class ClientIpStore(ClientIpWorkerStore):
   	def __init__(self, database: DatabasePool, db_conn, hs):
   	self.client_ip_last_seen = LruCache(
   	cache_name="client_ip_last_seen", max_size=50000
   	)
   	super().__init__(database, db_conn, hs)
   	# (user_id, access_token, ip,) -> (user_agent, device_id, last_seen)
   	self._batch_row_update = {}
   	self._client_ip_looper = self._clock.looping_call(
   	self._update_client_ips_batch, 5 * 1000
   	)
   	self.hs.get_reactor().addSystemEventTrigger(
   	"before", "shutdown", self._update_client_ips_batch
   	)

   So something like:

           if hs.config.run_background_tasks:
               self._client_ip_looper = self._clock.looping_call(
                   self._update_client_ips_batch, 5 * 1000
               )

   as well as the other bits in the constructor.
3. clokep commented on Jul 19, 2021

   

   clokep

   on Jul 19, 2021

   Member

   More actions

   It doesn't look like #8500 has any info on why this wasn't done.

   I believe it was because insert_client_ip is what caches the IP batch info (by writing to _batch_row_update). This gets called by the master process, so it isn't as simple as just saying "run this other code on the background worker" since...that worker will have the in-memory cache updated.

   You might be able to send those over replication, but that would be more invasive.

   👍React with 👍1Reacted by Andrew Morgan
4. anoadragon453 commented on Jul 28, 2021

   

   anoadragon453

   on Jul 28, 2021

   MemberAuthor

   More actions

   @clokep does that imply that IP addresses are only noted in the database for the main process, not any workers?
5. clokep commented on Jul 29, 2021

   

   clokep

   on Jul 29, 2021

   Member

   More actions

   @clokep does that imply that IP addresses are only noted in the database for the main process, not any workers?

   I think so but it has been a while since I looked at this. 😄
6. anoadragon453 commented on Jul 29, 2021

   

   anoadragon453

   on Jul 29, 2021

   MemberAuthor

   More actions

   In addition, there's a simple optimisation we could make to the function to likely make it a bit lighter on the database.

   Currently we iterate over the list of all IP entries and make one or (more often) 2 separate queries against the database server. This results in both a high amount of DB and CPU usage.

   synapse/synapse/storage/databases/main/client_ips.py

   Lines 494 to 524 in c0df6ba

   	for entry in to_update.items():
   	(user_id, access_token, ip), (user_agent, device_id, last_seen) = entry
   	self.db_pool.simple_upsert_txn(
   	txn,
   	table="user_ips",
   	keyvalues={"user_id": user_id, "access_token": access_token, "ip": ip},
   	values={
   	"user_agent": user_agent,
   	"device_id": device_id,
   	"last_seen": last_seen,
   	},
   	lock=False,
   	)
   	# Technically an access token might not be associated with
   	# a device so we need to check.
   	if device_id:
   	# this is always an update rather than an upsert: the row should
   	# already exist, and if it doesn't, that may be because it has been
   	# deleted, and we don't want to re-create it.
   	self.db_pool.simple_update_txn(
   	txn,
   	table="devices",
   	keyvalues={"user_id": user_id, "device_id": device_id},
   	updatevalues={
   	"user_agent": user_agent,
   	"last_seen": last_seen,
   	"ip": ip,
   	},
   	)

   One simple solution we could do is to simply execute a single batch query against the server containing all modifications. We could:

   • transform the numerous calls of simple_upsert_txn to a single simple_upsert_many_txn, which utilises txn.execute_batch.
   • simple_update_txn does not (yet) have a _many derivative. I'm not sure whether we'd want to create that for this purpose.
7. reivilibre

   self-assigned this

   on Mar 17, 2022
8. reivilibre

   closed this as completedin #12251on Apr 1, 2022