Run every suite this repo owns, not one spec of one - #259
Open
vitramir wants to merge 7 commits into
Open
Conversation
The workflow named a single file: E2E_SUITES: playwright E2E_PLAYWRIGHT_TEST_ARGS: test/e2e/organization-platform-usage.spec.ts So the repo that owns the suites ran one of its 101 specs and test files, and every other change to them landed on the word of a run that never executed them. Three jobs now cover the lot: 22 console specs, 18 across the two app frontends, 61 go-core test files. A job apiece rather than one for all three, because a VM boot is ten minutes and the suites together do not fit in a job that should stay under an hour -- and a red job then names the thing that broke. The tag differs per job because the suites read it in opposite directions. A tag becomes a --grep for playwright, and console-api, chat-api and gateway-api carry none, so filtering silently drops them: those jobs pass no tag. go-core with no tag runs smoke alone, so it names its services. playwright-chat-app opted out of a run that asks for nothing, alone among the playwright suites, which left it unreachable from a run of everything. It selects like its siblings now. go-terraform and go-agn-cli stay out. Both need a binary built elsewhere, and they run from the workflows that have it.
run-tests created an LLM provider and a model over the Gateway's HTTP API before every run, for the one suite that could not make its own. It cost 193 lines, two variables, and a step that had to be reachable and ready from the runner -- and it failed the whole `services` job on a 503 in the seconds after a rollout, before a single Go test had run. go-core has resolved its own model since the provisioning moved; this gives go-terraform the same. AGYN_MODEL_ID still wins when a caller has one, and a 503 is retried rather than ending the job, because by then it is one suite waiting rather than every suite blocked. E2E_GATEWAY_URL went with it: it existed to give the seeding an address the runner could reach, and nothing else read it.
The chat and tracing suites carry their own copies of the sign-in helper, and only the console's learned Dex. So on the bundle VM neither could sign in at all: 23 of the apps job's 27 specs failed waiting for a chat list or a no-organizations screen, on a page that was still Dex's login form. Same patch as the console's -- match the password field, fill it, submit -- and the same bundled member by default. Three copies of one helper is the actual problem here, and this is not the change that fixes that.
Asking for every service tag at once does not compile. go-core's helper files deliberately repeat each other behind mutually exclusive service tags -- two TestMains, four workload constants, fourteen names in all -- so the package stops building the moment two of them are on together. I started deduplicating and stopped: making a suite one package to suit a runner is the wrong way round, and there is no coverage that would catch a behaviour change while merging helpers that only look identical. A job per service instead, which is what every calling repo already does. A red job then names the service rather than the suite. smoke rides along, because the suite assumes it: main_test.go is built for nine service tags and calls helpers only a smoke-tagged file defines, so svc_k8s_runner alone does not compile. svc_egress_gateway is the one exception -- it repeats the k8s-runner workload constants, which smoke also carries -- so it runs alone. Every combination here was compiled before it was written down.
Two more calls the services answer with "cluster admin required" once the suite signs in as an ordinary member, and neither is what its spec is about. listUsers is not asserted on anywhere; it is how inviteMember finds the identity behind an address. registerRunner seeds the runners whose listing is the actual subject of runners.spec. Both go through the bootstrap token, like createUser.
Its sections are an overview of every organization, the user directory, cluster runners and the app catalog. An ordinary member has no route to any of them, so the test sat waiting for a sidebar that was never going to render. The organization sections in the same file stay on the member, which is who they are for.
The suite's setup posted a password grant to keycloak.keycloak.svc.cluster.local, which the platform stopped running when Dex replaced it. So the whole package failed its setup -- "e2e setup failed: setup credentials" -- rather than any test failing on its own account. Dex serves the same grant; the chart sets oauth2.passwordConnector to `local`, which is what enables it. The username becomes the address, because Dex looks a static user up by its email field and has nothing else. The rest holds: Dex stamps `iss` from its configured issuer, not from the address it was called on, so a token minted over plain in-cluster HTTP still verifies against a Media Proxy configured with the browser-facing one. Verified against a running VM, from a pod and through the ingress.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The workflow named a single file:
So the repo that owns the suites ran one of its 101 specs and test files. Every other change to them — including the sign-in rework that just landed — went in on the word of a run that never executed them.
Three jobs now cover the lot
consoleplaywrightappsplaywright-chat-app,playwright-tracing-appservicesgo-coreA job apiece rather than one for all three: a VM boot is ten minutes and the suites together do not fit under an hour, and a red job then names the thing that broke.
fail-fast: false, so one failure does not hide the others.Why the tag differs per job. The suites read it in opposite directions. For playwright a tag becomes a
--grep, andconsole-api.spec.ts,chat-api.spec.tsandgateway-api.spec.tscarry no tag — so any filter silently drops them. Those jobs pass none.go-coreis the reverse: with no tag it runssmokealone, so it names its services explicitly.One suite fix.
playwright-chat-appwas the only playwright suite whoseselectopted out when asked for nothing, which made it unreachable from a run of everything. It now selects like its siblings. Verified by simulating every suite'sselectagainst each tag set.Deliberately excluded:
go-terraformandgo-agn-cli. Both need a binary built elsewhere — the provider from terraform-provider-agyn, the CLI from an agn-cli release — and run from the workflows that have it. Without one they reach for a public registry that does not serve these providers, which is the failure console-app hit on the old bootstrap path.Expect this first run to be red. It is the first time 100 of these have executed against the VM at all, and the ordinary-member sign-in that landed in #258 has only ever been tried against one spec — I would expect several
403 cluster admin requiredamong them, each either a setup call that should use the bootstrap token or a spec that belongs onadminPage.