Fix docker image layer caching to avoid redundant docker building and transient connection exceptions.

[Yi Zhang (mszhanyi)]

Description

Improve docker commands to make docker image layer caching works.
It can make docker building faster and more stable.
So far, A100 pool's system disk is too small to use docker cache.
We won't use pipeline cache for docker image and remove some legacy code.

Motivation and Context

There are often an exception of

64.58 + curl https://nodejs.org/dist/v18.17.1/node-v18.17.1-linux-x64.tar.gz -sSL --retry 5 --retry-delay 30 --create-dirs -o /tmp/src/node-v18.17.1-linux-x64.tar.gz --fail
286.4 curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

Because Onnxruntime pipeline have been sending too many requests to download Nodejs in docker building.
Which is the major reason of pipeline failing now

In fact, docker image layer caching never works.
We can always see the scrips are still running

#9 [3/5] RUN cd /tmp/scripts && /tmp/scripts/install_centos.sh && /tmp/scripts/install_deps.sh && rm -rf /tmp/scripts
#9 0.234 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#9 0.235 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#9 0.235 /tmp/scripts/install_centos.sh: line 1: !/bin/bash: No such file or directory
#9 0.235 ++ '[' '!' -f /etc/yum.repos.d/microsoft-prod.repo ']'
#9 0.236 +++ tr -dc 0-9.
#9 0.236 +++ cut -d . -f1
#9 0.238 ++ os_major_version=8
....
#9 60.41 + curl https://nodejs.org/dist/v18.17.1/node-v18.17.1-linux-x64.tar.gz -sSL --retry 5 --retry-delay 30 --create-dirs -o /tmp/src/node-v18.17.1-linux-x64.tar.gz --fail
#9 60.59 + return 0
...

This PR is improving the docker command to make image layer caching work.
Thus, CI won't send so many redundant request of downloading NodeJS.

#9 [2/5] ADD scripts /tmp/scripts
#9 CACHED

#10 [3/5] RUN cd /tmp/scripts && /tmp/scripts/install_centos.sh && /tmp/scripts/install_deps.sh && rm -rf /tmp/scripts
#10 CACHED

#11 [4/5] RUN adduser --uid 1000 onnxruntimedev
#11 CACHED

#12 [5/5] WORKDIR /home/onnxruntimedev
#12 CACHED

###Reference
https://docs.docker.com/build/drivers/

[Changming Sun (snnn)]

The machine actually has a very big temp disk, but it needs to be manually initialized.

[Yi Zhang (mszhanyi)]

The machine actually has a very big temp disk, but it needs to be manually initialized.

I tried it looks 1ES will restore the setting when the VM image is loaded.
That is, I haven't find a solution to use nvme disk in 1ES A100 pool.
Maybe we need to wait 1ES to support it.

[Yi Zhang (mszhanyi)]

It'd best to add rel-1.19.0.
It obviously improved pipeline stability.
Without it, after running rel-1.19.0, the cache will not exist in ACR and it will run many unnecessary image rebuilding.