[ci] Move pip dependencies to docker images, add ninja / shellcheck

[driazati]

Following on from #10246, this moves the pip install-at-runtime deps to the docker image install so they are baked in. This adds some extra dependencies to support #10425 and make it so we can easily run things in parallel in CI scripts via GNU parallel and lint our bash scripts.

[areusch]

@driazati what's the plan for removing task_ci_setup.sh tho?

[areusch]

actually let's discuss that first before merging, as changes between the upstream index and what we pin can cause task_ci_setup.sh to start failing

[driazati]

@driazati what's the plan for removing task_ci_setup.sh tho?

once this lands and we update the docker images we can remove the pip installs from there

actually let's discuss that first before merging, as changes between the upstream index and what we pin can cause task_ci_setup.sh to start failing

the version pins are the same so the installs intask_ci_setup.sh should be a no-op once this is added

[areusch]

i think maybe we should change task_ci_setup.sh to not install those packages if they already exist. i can't remember exactly what happened here, but there was a CI problem before where a disagreement between the pinned version of the package in task_ci_setup.sh and the pinned version in the docker/install scripts caused CI to fail. i think maybe this was due to not specifying -U in pip install? anyway, it would be great to ensure that either we use task_ci_setup.sh or we use new docker image.

[areusch]

ok, discussed offline, since we are migrating all of these to the ci image, the main thing we need to avoid the state where one version of synr is installed in ci- images and another in $TVM_HOME/.local. since we are moving all of them to ci installation, we can just rm -rf $TVM_HOME/.local in task_ci_setup.sh when we bump the docker image version. we can revert the docker image with [skip ci] to avoid breakage.

[areusch]

@Mousius can you take another look here and merge if you are good with it?

[Mousius]

LGTM! Apologies for the delay!

Agree dependencies which are used as Python dependencies should come via pip. If they're not used as Python dependencies and there's a stable OS package available then we should favour the more robust OS way of delivering packages.