Make manifest cache size configurable and allow for disabling

[kris-gaudel]

Feature Request / Improvement

Discussion in #2325 revealed that the manifest cache leaks memory over time. On-going work to fix the leak is outlined in #2951, but having the ability to configure the cache size and disable it entirely would be beneficial to developers

[kevinjqliu]

Im thinking either set as env var or catalog properties. Leaning towards env var since this might be use in places other than the catalog route. No hard opinions though

cc @Fokko if you have any preferences here