CCDB: report stats about CCDB fetches / misses to DPL

[ktf]

No description provided.

[github-actions]

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1
async-2024-PbPb-apass2
async-2023-PbPb-apass5

[alibuild]

Error while checking build/O2/fullCI_slc9 for 7e8a2e5 at 2026-02-17 23:36:

http://madgraph.mi.infn.it//Downloads/HEPToolsInstaller/HEPToolsInstaller_V168.tar.gz
Now installing madanalysis5. Be patient...
/sw/BUILD/f582e4ec92cdbf97225eeb9801743861f47a6049/madgraph/HEPTools/HEPToolsInstallers/HEPToolInstaller.py:650: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _mg5_version = LooseVersion(out[0])
/sw/BUILD/f582e4ec92cdbf97225eeb9801743861f47a6049/madgraph/HEPTools/HEPToolsInstallers/HEPToolInstaller.py:543: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if tool_name in ['lhapdf6', 'lhapdf'] and MG5_version and MG5_version < LooseVersion("2.6.1"):
/sw/BUILD/f582e4ec92cdbf97225eeb9801743861f47a6049/madgraph/HEPTools/HEPToolsInstallers/HEPToolInstaller.py:298: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  ( lambda MG5version: MG5version < LooseVersion("2.6.1"),
Detected 'madanalysis5' missing dependency: 'zlib'. Will install it now.
Fetching data with command:
  wget --no-check-certificate http://www.zlib.net//zlib-1.3.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found
Could not download data at '['http://www.zlib.net//zlib-1.3.1.tar.gz', 'http://madgraph.phys.ucl.ac.be/Downloads//zlib-1.3.1.tar.gz', 'http://madgraph.physics.illinois.edu/Downloads/zlib-1.3.1.tar.gz']' because of:


�[1;34mDefault installation of Madanalys5 failed.�[0m
�[1;34mMG5aMC will now attempt to reinstall it with the options '--no_MA5_further_install --no_root_in_MA5'.�[0m
�[1;34mThis will however limit MA5 applicability for hadron-level analysis.�[0m
�[1;34mIf you would like to prevent MG5aMC to re-attempt MA5 installation, start MG5aMC with './bin/mg5_aMC --debug'.�[0m
Downloading the HEPToolInstaller at:
   http://madgraph.mi.infn.it//Downloads/HEPToolsInstaller/HEPToolsInstaller_V168.tar.gz
�[1;31mCommand "import /sw/BUILD/f582e4ec92cdbf97225eeb9801743861f47a6049/madgraph/install.dat" interrupted in sub-command:
"install MadAnalysis5" with error:
RecursionError : maximum recursion depth exceeded in comparison
Please report this bug on https://bugs.launchpad.net/mg5amcnlo
More information is found in 'MG5_debug'.
Please attach this file to your report.�[0m
Checking if MG5 is up-to-date... (takes up to 5s)
New Version of MG5 available! Do you want to update your current version? [�[4mn�[0m, y]
�[1mThe answer to the previous question is not set in your input file�[0m
�[1mUse n value�[0m
Update bypassed.
The next check for a new version will be performed in 7 days
In order to change this delay. Enter the command:
set auto_update X
Putting X to zero will prevent this check at anytime.
You can upgrade your version at any time by typing:
install update
quit
++ rm install.dat
++ rm HEPTools/ninja/ninja_install.log
++ rm HEPTools/oneloop/oneloop_install.log
++ rm HEPTools/collier/collier_install.log
++ rm HEPTools/madanalysis5/madanalysis5_install.log
rm: cannot remove 'HEPTools/madanalysis5/madanalysis5_install.log': No such file or directory

Full log here.

[alibuild]

Error while checking build/O2/fullCI_slc9 for 2209231 at 2026-02-18 04:54:

http://madgraph.phys.ucl.ac.be//Downloads/HEPToolsInstaller/HEPToolsInstaller_V168.tar.gz
Now installing madanalysis5. Be patient...
/sw/BUILD/f582e4ec92cdbf97225eeb9801743861f47a6049/madgraph/HEPTools/HEPToolsInstallers/HEPToolInstaller.py:650: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _mg5_version = LooseVersion(out[0])
/sw/BUILD/f582e4ec92cdbf97225eeb9801743861f47a6049/madgraph/HEPTools/HEPToolsInstallers/HEPToolInstaller.py:543: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if tool_name in ['lhapdf6', 'lhapdf'] and MG5_version and MG5_version < LooseVersion("2.6.1"):
/sw/BUILD/f582e4ec92cdbf97225eeb9801743861f47a6049/madgraph/HEPTools/HEPToolsInstallers/HEPToolInstaller.py:298: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  ( lambda MG5version: MG5version < LooseVersion("2.6.1"),
Detected 'madanalysis5' missing dependency: 'zlib'. Will install it now.
Fetching data with command:
  wget --no-check-certificate http://www.zlib.net//zlib-1.3.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found
Could not download data at '['http://www.zlib.net//zlib-1.3.1.tar.gz', 'http://madgraph.phys.ucl.ac.be/Downloads//zlib-1.3.1.tar.gz', 'http://madgraph.physics.illinois.edu/Downloads/zlib-1.3.1.tar.gz']' because of:


�[1;34mDefault installation of Madanalys5 failed.�[0m
�[1;34mMG5aMC will now attempt to reinstall it with the options '--no_MA5_further_install --no_root_in_MA5'.�[0m
�[1;34mThis will however limit MA5 applicability for hadron-level analysis.�[0m
�[1;34mIf you would like to prevent MG5aMC to re-attempt MA5 installation, start MG5aMC with './bin/mg5_aMC --debug'.�[0m
Downloading the HEPToolInstaller at:
   http://madgraph.phys.ucl.ac.be//Downloads/HEPToolsInstaller/HEPToolsInstaller_V168.tar.gz
�[1;31mCommand "import /sw/BUILD/f582e4ec92cdbf97225eeb9801743861f47a6049/madgraph/install.dat" interrupted in sub-command:
"install MadAnalysis5" with error:
RecursionError : maximum recursion depth exceeded in comparison
Please report this bug on https://bugs.launchpad.net/mg5amcnlo
More information is found in 'MG5_debug'.
Please attach this file to your report.�[0m
Checking if MG5 is up-to-date... (takes up to 5s)
New Version of MG5 available! Do you want to update your current version? [�[4mn�[0m, y]
�[1mThe answer to the previous question is not set in your input file�[0m
�[1mUse n value�[0m
Update bypassed.
The next check for a new version will be performed in 7 days
In order to change this delay. Enter the command:
set auto_update X
Putting X to zero will prevent this check at anytime.
You can upgrade your version at any time by typing:
install update
quit
++ rm install.dat
++ rm HEPTools/ninja/ninja_install.log
++ rm HEPTools/oneloop/oneloop_install.log
++ rm HEPTools/collier/collier_install.log
++ rm HEPTools/madanalysis5/madanalysis5_install.log
rm: cannot remove 'HEPTools/madanalysis5/madanalysis5_install.log': No such file or directory

Full log here.

[ktf]

Seems to always publish "0" for all metrics. I am investigating why that would happen.

[ktf]

Of course, actually running a test which uses CCDB would probably help... Works as expected on hyperloop. Full CI OK. Merging.

[ktf]

https://alimonitor.cern.ch/train-workdir/tests/0061/00618501/performanceMetrics.json for reference.

[costing]

Ciao Giulio,

That's great, thanks a lot!

I see ccdb-cache-* entries in all devices and a handful have !=0 values . So does that mean that we have to target those tasks to switch to using DPL ?

Would it be possible to have a summary of these metrics at a workflow level?

Cheers,

.costin

[ktf]

I see ccdb-cache-* entries in all devices and a handful have !=0 values . So does that mean that we have to target those tasks to switch to using DPL ?

yes, correct. of course DPL usage will not improve things for objects which are read by only one device.

Would it be possible to have a summary of these metrics at a workflow level?

Can't we do it by post processing the jsons? I have no infrastructure in DPL for that right now.