docker buildx bake should apply env variable from .env

[vinverdy]

docker buildx should read .env file like compose. Kudos

docker-compose.yml

version: '3.6'

services:
    ssr:
        build:
            context: .
            target: ssr
            args:
                COMMIT_SHA: ${COMMIT_SHA}

.env

COMMIT_SHA=a06baf1a

docker-compose config

services:
  ssr:
    build:
      args:
        COMMIT_SHA: a06baf1a
      context: 
      target: ssr
version: '3.6'

docker buildx bake --print

{
   "target": {
      "ssr": {
         "context": ".",
         "dockerfile": "Dockerfile",
         "args": {
            "COMMIT_SHA": ""
         },
         "target": "ssr"
      }
   }
}

[tonistiigi]

@tiborvass @FernandoMiguel wdyt? I remember discussing this case some time ago with at least one of you.

[FernandoMiguel]

I remember opening this
#111

but yeah, there is expectation for those values to be read...

[ktns]

If you think it's a burden to have an docker-compose.yml parser in buildx for this, what about using docker-compose config command for expanding environment variables? It outputs the content with all environment variables substituted.

Maybe it should not be a default behavior, but I'll be happy if there is such an option.

[crazy-max]

If it's the right direction, #489 can be tested if you want: https://github.com/docker/buildx/suites/1696585811/artifacts/31897347

[aeaton-overleaf]

I'm working around this with env $(cat .env | grep -v "#" | xargs) docker buildx … but it would be useful to be able to specify --env-file=.env as an option even if it's not on by default.

[crazy-max]

Fixed by #541

[zachary-povey]

Hi 👋 - I think this should be re-opened.

#541 allows specifying this kind of file for HCL configurations but doesn't give any options for people still using docker-compose. The .env file is still not read automatically and passing it through using -f also does not work nor does using the default docker-bake.override.hcl file.

@tonistiigi mentions this in the issue:

The HCL attributes are not 100% compatible with env file format so I think for compose support we still should have a fallback, and maybe still automatically load it if you parse compose. Possibly we can just use the new compose library that does that automatically on parsing compose. For everything else, you can use HCL/JSON attributes.

[trinitronx]

@crazy-max I'd tend to agree with @zachary-povey given the caveat of @tonistiigi's observation that an env.hcl file is not 100% compatible to .env (although in theory, a subset of HCL key="value" pairs could suffice to be valid in both contexts).

Feature Idea

For example, the following commands could be made to work functionally equivalent for most general variable substitution purposes given the same .env file:

# docker-compose.buildx.yml: Could use x-bake extension to docker-compose.yaml syntax
# with matching definitions in an HCL equivalent file: docker-buildx-bake.hcl
docker buildx bake --file docker-compose.buildx.yml --file .env --print multi-platform-demo
docker buildx bake --file docker-buildx-bake.hcl    --file .env --print multi-platform-demo
docker-compose -f docker-compose.buildx.yml  config

Input Files:

.env:

# vim: syntax=hcl filetype=hcl
# Note: Quoted k="v" syntax supports both docker-compose & docker buildx bake
#       Try to keep this file simple w/o advanced HCL constructs
FOO="bar"

docker-compose.buildx.yml:

# vim: syntax=yaml filetype=yaml
version: "3.4"

services:
  multi-platform-demo:
    build:
      context: .
      # dockerfile: ./Dockerfile
      # args:
      target: release
      x-bake:
        tags:
          - test-tag:${FOO}
        platforms:
          - linux/amd64
          - linux/arm64
          - linux/ppc64le
          - linux/riscv64
          - linux/s390x
          - linux/386
          - linux/arm/v7
          - linux/arm/v6
        cache-from:
          - type=local,src=./cache/multi-platform-demo
        cache-to:
          - type=local,dest=./cache/multi-platform-demo
        output: type=registry,push=false,name=trinitronx/multi-platform-demo

# vim: syntax=hcl filetype=hcl
variable "FOO" { default = "foo" }

target "multi-platform-demo" {
  context = "."
  # dockerfile = "./Dockerfile"
  # args = {}
  
  target = "release"
  # BuildX (e.g. docker-compose.buildx.yml x-bake) {
    tags = [ "test-tag:${FOO}" ]
    platforms = [
      "linux/amd64",
      "linux/arm64",
      "linux/ppc64le",
      "linux/riscv64",
      "linux/s390x",
      "linux/386",
      "linux/arm/v7",
      "linux/arm/v6"
    ]

    cache-from = [ "type=local,src=./cache/multi-platform-demo" ]
    cache-to = [ "type=local,dest=./cache/multi-platform-demo" ]
    output = [ "type=registry,push=false,name=trinitronx/multi-platform-demo" ]
  # } end x-bake equivalent
}

Expected Result

The first two buildx commands could return the same JSON, while the docker-compose one performs the same variable substitution given the .env file and outputs the usual YAML format.

Actual Result

diff -Naur \
  <(docker buildx bake --file docker-compose.buildx.yml --file .env --print multi-platform-demo) \
  <(docker buildx bake --file docker-buildx-bake.hcl    --file .env --print multi-platform-demo)

WARN[0000] The "FOO" variable is not set. Defaulting to a blank string.
[+] Building 0.0s (0/0)
[+] Building 0.0s (0/0)
--- /dev/fd/63	2022-02-18 20:44:16.000000000 -0700
+++ /dev/fd/62	2022-02-18 20:44:16.000000000 -0700
@@ -11,7 +11,7 @@
       "context": ".",
       "dockerfile": "Dockerfile",
       "tags": [
-        "test-tag:"
+        "test-tag:bar"
       ],
       "cache-from": [
         "type=local,src=./cache/multi-platform-demo"

The current state of things is that the docker-compose bake file format is not in feature parity with the HCL file format when it comes to the .env file. Given that YAML and JSON could be considered as subsets of HCL, there will never be 100% feature parity. So we shouldn't expect 100% parity given the nature of the two languages. However, for simple key = "value" type expressions that are valid in both, it would make sense to support this (at least for .env files).

Currently, the command alternative: docker buildx bake --file docker-compose.buildx.yml --file .env doesn't actually evaluate the .env variables file at all.

Workaround

A hack-ish workaround that currently appears to work is to simply export the variables:

eval $(sed -e '/^#/d' -e 's/^/export /' -e 's/$/;/' .env) ;
docker buildx bake --file docker-compose.buildx.yml --print multi-platform-demo

This appears to work for simple variable substitutions, which get printed out in the buildx bake JSON output.

diff -Naur <(eval $(sed -e '/^#/d' -e 's/^/export /' -e 's/$/;/' .env) ; \
             docker buildx bake --file docker-compose.buildx.yml          --print multi-platform-demo) \
           <(docker buildx bake --file docker-buildx-bake.hcl --file .env --print multi-platform-demo)
[+] Building 0.0s (0/0)
[+] Building 0.0s (0/0)
## No difference in output now... thanks to exporting all variables in .env

[crazy-max]

@trinitronx This should be fixed by #905 (not released yet but you can build from source)

[cdupont]

I tried buildx v0.8.0 but this issue doesn't seem to be fixed.

docker-compose.yml:

version: "3.9"

services:
  foo:
    image: foo:$TAG

.env:

TAG=bar

$ docker buildx bake
[+] Building 0.0s (0/0)                                                                                                                        
WARNING: The "TAG" variable is not set. Defaulting to a blank string.

[olidroide]

same to me @cdupont with dockerx v0.8.0 also v0.8.1

with docker-compose works perfect taking .env file from same directory.

[eddietisma]

I also encountered this issue using v.0.8.2. We're not ready to migrate to .hcl yet. Re-open?

[marcellodesales]

Sam problem here... The env var is not applied by default just like docker-compose does...