evaluate memory usage optimization potential for borg

[ThomasWaldmann]

see there: https://habr.com/en/post/458518/

[ThomasWaldmann]

itamar gave a nice talk at europython 2021:

https://pythonspeed.com/europython2021

https://pythonspeed.com/memory

[ThomasWaldmann]

note: there were significant changes recently in master branch, influencing the biggest memory allocations in borg:

• no repo index anymore, each object stored separately, found via id in path/filename
• chunkindex: using borghash now instead of old hashindex C code, less overhead
• files cache: new "compression" function, abbreviating the chunks list entries using kv array indexes into chunkindex kv array

[ThomasWaldmann]

Note: In borg 1.4.4 (and also commits to 1.1-maint and 1.2-maint branches) a memory leak in hashindex was fixed.

[ThomasWaldmann]

changes in master branch since the packs work started:

the chunks index now also has pack_id, obj_offset, obj_size infos (thus, the information that used to be in the repo index in borg 1.x is now in the chunks index on the client side). That increases the per entry memory needs by 40B.