probes do not treat 404 status as error

[klauss42]

If I configure probes I expected that mint fails if probes return an error. I would expect that both 40x or 50x status codes for probes should let mint stop processing and return an error.

E.g.

mint slim \
  --target nginx:latest \
  --tag nginx:latest-slim \
  --expose 80 \
  --show-clogs \
  --http-probe-off \
  --http-probe-cmd / \
  --http-probe-cmd /invalid

results in

...
cmd=slim state=http.probe.running
cmd=slim info=http.probe.ports count='1' targets='32783'
cmd=slim info=http.probe.commands count='2' commands='GET /,GET /invalid'
cmd=slim info=http.probe.call endpoint='http://127.0.0.1:32783/invalid' attempt='1' error='none' time='2025-01-27T10:05:53Z' status='404' method='GET'
cmd=slim info=http.probe.call attempt='1' error='none' time='2025-01-27T10:05:53Z' status='200' method='GET' endpoint='http://127.0.0.1:32783/'
cmd=slim info=http.probe.summary total='2' failures='0' successful='2'
cmd=slim state=http.probe.done
cmd=slim info=event message='HTTP probe is done'
cmd=slim state=container.inspection.finishing
...

mint reports 0 failures and continues building the image. In my case I want mint to fail, so that the build pipeline fails because something seems to be wrong with the target image resulting in the 404 (or 500) error.

Did I misunderstand or misuse something?

[kcq]

@klauss42 The probes are not traditional tests and they don't test for correctness in terms of what the response looks like. It's more about generating interactions with the applications and from that perspective it also benefits from going outside of the happy path :-) In a way, it's a bit like an input fuzzer... However, in some cases you do want a specific behavior and, in some cases, that specific response behavior is needed to get to certain parts of the application.

Either way, you might find the --http-probe-fail-on-status-5xx flag valuable because it gives you half of what you are asking for here :-)

The 4xx responses are a bit trickier when it comes to the default behavior. The most likely solution there will be to introduce some kind of 'expected response code' as one of the probe params. The probes will need to be scoped better too because, by default, the probes are executed against all available ports and if you have two ports (one for the main interface and another one for the admin interface, for example) one of them might return 404 (because the given endpoint isn't implemented there) and another might return 200 (e.g., it can be an admin interface only endpoint).

Somewhat related... In some cases the response codes are different if the app is not fully initialized. Right now waiting for the app to fully initialize involves adding a wait time, which may not be always the same. The upcoming enhancements will make it possible to specify a 'readiness' endpoint probe that will simplify that setup.

[klauss42]

Thanks for your answer.

Let me describe my problem: I am using mint to build a slim image of a quite huge machine learning model. Using the probes I am calling all relevant entrypoints of the image which works fine in most case. But sometimes startup of the container or initialization takes a bit longer and I expected that mint (and the build pipeline) would fail in these cases. But instead mint finishes the probes (with some probes on error) but continues building the slim image. I ended up with a slim image that did not work at runtime because mint removed stuff from the image that were important at runtime.
I added the fail-on-status-5xx option, added wait time and tried the error count option, but it feels a bit unreliable to me.

Any ideas how to handle this?

[kcq]

@klauss42 Very helpful context! I can quickly add --fail-on-status-4xx as mostly copy and paste of --fail-on-status-5xx as a way to unblock you there :-) Personally I'd want to leverage the other upcoming enhancements including expected response code in custom probes, http probe readiness endpoint to start probing only after that endpoint returns ok/200, and log-based wait flags, but adding all those will take longer with everything I have on my plate at the moment.