[v6r22] Pilot Submission statistics is counted by SiteDirector based on Accounting System

[yujikato]

BEGINRELEASENOTES

WMS and Accounting
NEW: Statistics of Pilot submission is sent to Accounting System by SiteDirector.

ENDRELEASENOTES

[zmathe]

I have a general comment. This is not accounting, I would propose to add it to the MonitoringSystem. Probably, that time when you developed this accounting type, the monitoring did not exist. It is easy to move this to the MonitoringSystem, will not require lot of work, because both systems are almost identical.

[zmathe]

please move RCSID before class declaration( line 7 ) and it must be:

__RCSID__ = "$Id$"

[zmathe]

maxValue is not used:

dataDict, _ = self._divideByFactor(dataDict, granularity)

[zmathe]

maxValue is not used here as well

[chaen]

@zmathe I think it's fine even if it is using the Accounting system.
This PR targets v6r22 so ES is not yet mandatory (from v7 we said)
And BelleII uses it, so it's a good moment to have it in vanilla DIRAC too.
The migration to Monitoring system is quite straight forward indeed, and I seem to remember we agreed with @yujikato to do it in a second step

[yujikato]

@zmathe thank you very much for your comments.
As Belle II has not launched Monitoring System, it is not easy for me to move immediately.
After we have monitoring system, I can also work on it.

I fixed other comments.

[fstagni]

Hi @yujikato and thanks for this. Questions:

• do you use this in production in Belle2 or would this be something new for you too?
• Did you see any performance degradation? The SiteDirector isn't already the fastest of the agents
• We have deployed several SiteDirectors in LHCb. Each treat a few sites, but some only treat a few CEs from a certain site. These cases are covered, right?

[fstagni]

Can you add a flag to disable this reporting?

[yujikato]

Thank you for questions and sorry for my late reply due to summer holiday.

* do you use this in production in Belle2 or would this be something new for you too?

It is now running at the BelleDIRAC certification server but not at production environment.
It will be implemented soon.

* Did you see any performance degradation? The SiteDirector isn't already the fastest of the agents

The commits to the AccountingSystem happens for each Site-CE-Queue combination,
not for each pilot jobs.
So the number of commits is much smaller than that for PilotAccounting
(also, there is no additional query to retrieve information).
In my environment, the execution time does not increase more than a few %.

* We have deployed several SiteDirectors in LHCb. Each treat a few sites, but some only treat a few CEs from a certain site. These cases are covered, right?

Is it done by "CEs" option? If so, yes, these case can be is covered.

[yujikato]

Can you add a flag to disable this reporting?

It is done.

[fstagni]

This one looks OK for merging, but I agree with @zmathe that eventually it should be moved from Accounting to Monitoring. I created a task for that: #4226

[fstagni]

@zmathe you have "requested changes", would you review OK now?