This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
500 errors with "duplicate key value" on remote_media_cache_thumbnails when dynamic_thumbnails enabled #2182
Description
Error is at the end of this issue (it's a long one).
The following URLs produce similar errors to the one included here:
- https://t2bot.io/_matrix/media/v1/thumbnail/matrix.org/HtRqOXgQyGkcePuSgsuBrLcx?width=128&height=128&method=crop
- https://t2bot.io/_matrix/media/v1/thumbnail/matrix.org/ARfYZMTdGlRCAGXurhYSZGwT?width=128&height=128&method=crop
- https://t2bot.io/_matrix/media/v1/thumbnail/matrix.org/pFUfshTtyiLsZAODtmHOOLop?width=128&height=128&method=crop
There may be more, however those 3 are easily noticed.
Changing the method to scale does not result in an error. This does not appear to affect local media.
I tried to delete the files and records from the database in varying combinations (delete files & delete records, only delete records, only delete files) in various running states of synapse (running, shut down, restart after). I tried various combinations of deleting data from the two tables (from both remote_media_cache and remote_media_cache_thumbnails, only _thumbnails, only _cache). No luck.
The error continues and does not go away. If I'm missing a step, please let me know. Otherwise, this appears to be a very specific and strange bug.
Synapse version: v0.20.0
2017-05-02 20:35:23,067 - synapse.access.http.8008 - 59 - INFO - GET-753- <IP REDACTED> - 8008 - Received request: GET /_matrix/media/v1/thumbnail/matrix.org/ARfYZMTdGlRCAGXurhYSZGwT?width=128&height=128&method=crop
2017-05-02 20:35:23,068 - synapse.util.async - 201 - INFO - GET-753- Acquired linearizer lock 'media_remote' for key ('matrix.org', 'ARfYZMTdGlRCAGXurhYSZGwT')
2017-05-02 20:35:23,079 - synapse.util.async - 208 - INFO - GET-753- Releasing linearizer lock 'media_remote' for key ('matrix.org', 'ARfYZMTdGlRCAGXurhYSZGwT')
2017-05-02 20:35:23,083 - synapse.metrics - 162 - INFO - - Collecting gc 0
2017-05-02 20:35:23,106 - synapse.rest.media.v1.thumbnailer - 93 - INFO - - Stored thumbnail in file u'/root/synapse/media_store/remote_thumbnail/matrix.org/Sg/El/RAnfzhTBOJHxWANIWpBP/128-128-image-png'
2017-05-02 20:35:23,118 - synapse.metrics - 162 - INFO - - Collecting gc 0
2017-05-02 20:35:23,120 - synapse.http.server - 139 - ERROR - GET-753- Failed handle request synapse.rest.media.v1.thumbnail_resource._async_render_GET on <synapse.rest.media.v1.thumbnail_resource.ThumbnailResource instance at 0x7f3e179dabd8>: <XForwardedForRequest at 0x7f3dcf04b6c8 method=GET uri=/_matrix/media/v1/thumbnail/matrix.org/ARfYZMTdGlRCAGXurhYSZGwT?width=128&height=128&method=crop clientproto=HTTP/1.0 site=8008>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/synapse/http/server.py", line 118, in wrapped_request_handler
yield request_handler(self, request)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/local/lib/python2.7/dist-packages/synapse/rest/media/v1/thumbnail_resource.py", line 71, in _async_render_GET
width, height, method, m_type
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/local/lib/python2.7/dist-packages/synapse/rest/media/v1/thumbnail_resource.py", line 194, in _select_or_generate_remote_thumbnail
desired_height, desired_method, desired_type
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/local/lib/python2.7/dist-packages/synapse/rest/media/v1/media_repository.py", line 318, in generate_remote_exact_thumbnail
t_width, t_height, t_type, t_method, t_len
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 406, in _simple_insert
self._simple_insert_txn, table, values,
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 319, in runInteraction
inner_func, *args, **kwargs
File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/usr/local/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 307, in _runWithConnection
compat.reraise(excValue, excTraceback)
File "/usr/local/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 298, in _runWithConnection
result = func(conn, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 313, in inner_func
func, *args, **kwargs
File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 242, in _new_transaction
r = func(txn, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 426, in _simple_insert_txn
txn.execute(sql, vals)
File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 80, in execute
self._do_execute(self.txn.execute, sql, *args)
File "/usr/local/lib/python2.7/dist-packages/synapse/storage/_base.py", line 110, in _do_execute
sql, *args
IntegrityError: duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"
DETAIL: Key (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type)=(matrix.org, ARfYZMTdGlRCAGXurhYSZGwT, 128, 128, image/png) already exists.
Activity
jkolo commentedon Jul 17, 2017
This is the reason I made pull request #1946
Half-Shot commentedon Apr 4, 2018
This continues to bite me all the time on riot-android and the toasts that keep popping up are getting annoying.
TildeSlashC0re commentedon Apr 12, 2018
I can confirm that I experience the same bug!
Seems to only apply to specific remote thumnails.
EDIT:
during stopped state, then restarting did the trick for me!
Half-Shot commentedon Jun 1, 2018
Still experiencing this. It's a pretty awful experience for some avatars to load and some to fail.
[-]Cannot get cropped thumbnail for some media[/-][+]Cannot generate thumbnails for some media[/+]tleydxdy commentedon Jun 13, 2018
Same in 0.31.1
ananace commentedon Jun 29, 2018
The issue can be solved without doing full table migrations - at least on PostgreSQL, technically without even requiring downtime.
zzottel commentedon Jul 5, 2018
Just had the same after enabling dynamic thumbnails on my Synapse.
Can confirm that the SQL posted by ananace helps, but what is the consequence? That now multiple thumbs of the same size are allowed, right?
But why does Synapse even try to create its own dynamically if there's already one present for the desired resolution?
TildeSlashC0re commentedon Sep 8, 2018
Dropping by to confirm that @ananace workaround did the trick!
console-cowboy commentedon Sep 20, 2018
This remains an issue as of the 0.33.4 Release. The @ananace workaround provides the relief others have mentioned though.
ArchangeGabriel commentedon Feb 6, 2019
Could some dev comment on @ananace workaround consequences? Should we just do that?
msrd0 commentedon Feb 24, 2019
Still an issue on 0.99.1.1, @ananace's workaround still works. Would be nice to know how that affects my server though. Does that mean I store a new thumbnail every time a client asks for one?
richvdh commentedon Mar 22, 2019
Another report of this at #4915. Logs are:
13 remaining items
richvdh commentedon Sep 8, 2020
fixed by #7124.
f0x52 commentedon Oct 12, 2020
I'm still getting these errors in v1.21.0 which afaik includes that commit?
richvdh commentedon Oct 12, 2020
sigh. Please could you share the logs, with the stack trace?
f0x52 commentedon Oct 13, 2020
deepbluev7 commentedon Oct 13, 2020
It seems like the migrations didn't delete the duplicate entries. The following two queries should fix that, if you run into those issues with the background update:
They delete the duplicates and should allow the migrations to pass, although I'm not sure, if that has any consequences.
richvdh commentedon Oct 13, 2020
@deepbluev7 I'm confused. I thought these exceptions were happening because the existing index (
ADD CONSTRAINT local_media_repository_thumbn_media_id_thumbnail_width_thum_key UNIQUE (media_id, thumbnail_width, thumbnail_height, thumbnail_type)) was too strict. How can people have duplicates which prevent them running the migration?That doesn't seem to be @f0x52's problem, anyway, because he's reporting an error with the new constraint. @f0x52: is it possible that you are just seeing two requests for the same thumbnail at the same time, and they are racing? I think that would be #6755, which is a somewhat different problem :/
deepbluev7 commentedon Oct 13, 2020
@richvdh It seems their databases contains duplicates, which is why the migration fails. They may have deleted the index manually or something, not sure, but I did verify, that the db contains duplicates and deleting them fixes the issue. At least running the above query seemed to fix it for both people reporting the issue. But yeah, that seems to not apply to @f0x52's issue. No idea about that one.
richvdh commentedon Oct 13, 2020
If others have a problem with the migration, I'd like to know what their database schema looks like.
For now I'm going to assume this is correctly fixed, and @f0x52 is seeing #6755.
alturiak commentedon Oct 13, 2020
@deepbluev7 @richvdh
That is correct, I did delete the index as proposed as a workaround at #2182 (comment), which probably lead to the duplicates in the first place. They wouldn't probably appear in unmodified installations.
richvdh commentedon Oct 13, 2020
right, yes, people who have manually modified their database schema will now need to manually fix the resulting damage :).
532910 commentedon Nov 19, 2020
It looks like this is fixed. Anyway I don't see this error more.
Ezwen commentedon Mar 12, 2021
Hi there, I've just noticed I have errors in my synapse 1.29.0 logs, such as:
Is it related to this issue?