[CI] Does anyone feel that out CI is becoming slower and slower?

[jcf94]

The recent PRs trend to spend more time on passing the CI, looks like mainly cost by waiting for the machine resources.

[tqchen]

Thanks @jcf94 . There was an outage that blocks the CI during the weekend which is being worked on, we now recovered some of the GPU nodes and will work to recover all of them.

[tqchen]

The CI nodes are recovered. There are more ongoing effort to reduce the CI running time.

Let us move to more concrete threads for improving the CI test times