Remote containers >= 0.238.2 breaks proxy usage

[sveint]

• VSCode Version: 1.69.2
• Local OS Version: Ubuntu 22.04
• Remote OS Version: RHEL 8.6, Docker version 20.10.7
• Remote Extension/Connection Type: SSH + Containers

Steps to Reproduce:

1. Create a devcontainer.json with a Dockerfile including RUN steps depending on internet access.
2. Use Reopen in container on a machine that only allows internet via proxy.
3. Building the image times out on RUN step relying on internet access.

Upgrading the remote containers extension completely broke our development workflow, as our development server has a mandatory proxy for internet access. After debugging for a bit, I think I've narrowed it down to usage of docker buildx build rather than the normal docker build. In both cases we could use Buildkit for building, so I'm not sure why this change is necessary. Using docker build with Buildkit enabled works, as Docker uses the system-wide Docker proxy configuration that is already in place, even with Buildkit.

I've googled extensively and from what I can see there is currently no workaround possible for making Buildkit pick up a systemwide proxy configuration via docker buildx build. One can add parameters or a config to Buildkit on the docker buildx create command, but that is not used in the vscode workflow as far as I can see. I don't think we can configure that one either in devcontainer.json.

As far as I understand, remote container extension is now completely broken for all users that has to use a proxy for internet access. Downgrading to 0.234.0 fixes the issue, but I'm concerned how long that version will be supported.

Is it a hard requirement to switch to docker buildx build rather than DOCKER_BUILDKIT=1 docker build or can it be reverted until a workaround is in place?

[sveint]

Christof Marti (@chrmarti) Hi, sorry for bumping this, but do you have any workaround for us on this one? Our workflow is still broken after the extension got upgraded and I'm concerned how long we can stay on 0.234.0.

[chrmarti]

We are tracking this on the CLI side as devcontainers/cli#185. No ETA yet, sorry.

[sveint]

We are tracking this on the CLI side as devcontainers/cli#185. No ETA yet, sorry.

Christof Marti (@chrmarti) Thanks for your reply. Apologies if I'm misunderstanding, but to me it seems like that issue is unrelated (?). My issue is with how Buildkit itself is invoked, not using nodejs to download external resources. I.e. it's the RUN step building a Dockerfile that is failing.

I suspect the devcontainers repo referenced will also encounter the same problem when building an image from scratch (even with a fix for the issue you referenced applied), but I didn't dig too deep into the code so I might be mistaken.

That said, I see there's a fix coming in devcontainers/cli, so I'm very happy if I'm wrong.

[eitsupi]

FYI, I use Devcontainer on a daily basis behind a proxy and have no proxy issues.
I am just doing the proxy configuration to the Docker daemon and containers as is commonly done.
(The problems I have been facing lately due to corporate proxies etc. are #6961 and #7150)

[sveint]

FYI, I use Devcontainer on a daily basis behind a proxy and have no proxy issues. I am just doing the proxy configuration to the Docker daemon and containers as is commonly done. (The problems I have been facing lately due to corporate proxies etc. are #6961 and #7150)

eitsupi Thank you, that's promising. What Docker version are you using? Also, what proxy configuration have you done?

[Jondeen]

Christof Marti (@chrmarti) Thanks for your reply. Apologies if I'm misunderstanding, but to me it seems like that issue is unrelated (?). My issue is with how Buildkit itself is invoked, not using nodejs to download external resources. I.e. it's the RUN step building a Dockerfile that is failing.

I suspect the devcontainers repo referenced will also encounter the same problem when building an image from scratch (even with a fix for the issue you referenced applied), but I didn't dig too deep into the code so I might be mistaken.

That said, I see there's a fix coming in devcontainers/cli, so I'm very happy if I'm wrong.

To elaborate on what Svein Tore Koksrud Seljebotn (@sveint) said:
The problem at hand here is not the VSCode container plugin being unable to communicate through its Node API, but the lack of integration of system wide proxy definitions from buildx. While docker build pulls in the proxy definitions in ~/.docker/config.json, docker xbuild has no such inherent environment inheritance. Additionally, there is an issue with userns not being correctly forwarded (which matters particularly with rootless setups) in buildx embedded in Docker <v20.10.13.

One solution is to setup a dedicated proxy-aware buildx builder to be used for VSCode image building, with something akin to:

proxy="proxy-host:proxy-port"
docker buildx create --use \
    --platform linux/amd64 \
    --platform linux/amd64/v2 \
    --platform linux/amd64/v3 \
    --platform linux/amd64/v4 \
    --platform linux/386 \
    --driver-opt env.HTTP_PROXY="$proxy" \
    --driver-opt env.HTTPS_PROXY="$proxy" \
    --driver-opt env.http_proxy="$proxy" \
    --driver-opt env.https_proxy="$proxy"

*Catch-all capitalisation, not including proxy-ignore statements

As mentioned requires Docker version >20.10.13 (which additionally corrects buildx handling of rootless setups, through correct forwarding of the --userns argument). Alternatively, buildx can be bumped in the user environment by:
id=$(docker create docker/buildx-bin /) && docker cp $id:/buildx ~/.docker/cli-plugins/docker-buildx && docker rm -v $id

Fixing this on the VSCode side would require there to be an option of disabling docker buildx and keep the old docker build-way of doing things.

I hope this clarifies things.

[eitsupi]

Svein Tore Koksrud Seljebotn (@sveint) Only daemon settings that can be edited with the sudo systemctl edit docker command and ~/.docker/config.json.
I am using Docker CE on Ubuntu and the version is the same as yours.

This is the first time I have heard of buildx ignoring ~/.docker/config.json I rather use docker buildx bake as an alternative of docker compose build because compose V2 ignores ~/.docker/config.json. (docker/compose#8797 (comment))

[Jondeen]

eitsupi Could you confirm that this is a setup using rootless docker?

[eitsupi]

Jonas Daniel (@Jondeen) Sorry, I have never used rootless mode, so if you mean that it is a problem specific to rootless mode, that makes sense for me.