Skip to content

PACKIT-5037: Enable Packit CI - #627

Merged
mfocko merged 13 commits into
packit:mainfrom
mfocko:ci/enable-packit
Jul 10, 2026
Merged

PACKIT-5037: Enable Packit CI#627
mfocko merged 13 commits into
packit:mainfrom
mfocko:ci/enable-packit

Conversation

@mfocko

@mfocko mfocko commented Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

TODO

  • Add token for commit triggers…
  • Drop the debugging commits
  • Reword commits where necessary

Follow-ups:

  • Decide what to do about requesting specific VMs (dropped in c5d7bf2); this has been disabled as it doesn't work in the way it was intended and I have not been able to find in docs alternative with this specific purpose.
  • Decide whether it makes sense to invest into branching of the plan for local execution. There is a possibility to customize the tmt plan to run locally (as everything is tucked away in containers anyways). I am not able to run this in VM on macOS as I have not packaged the needed tmt deps for it :)
    I think I could adjust the provisioning for running the tmt tests locally to actually run locally 🤔 prepare clones the secrets and Jiras/repos from GitLab (needs the token exposed to env though…) and then everything is wrapped in the podman-compose, so locally it doesn't make sense to have one more layer of abstraction in form of VM (provisioned by tmt), OTOH it clones the secrets/Jiras/repos one more time :/

@mfocko mfocko self-assigned this Jun 25, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jun 25, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Packit target environments to fedora-latest-stable and adds an encrypted GITLAB_CI_TOKEN to the Testing Farm configuration. The reviewer noted that the GITLAB_CI_TOKEN secret is incorrectly formatted as a list, which will cause Testing Farm to fail to parse the configuration or break the git clone command. It should be defined as a single string instead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread .testing-farm.yaml
@mfocko
mfocko force-pushed the ci/enable-packit branch 5 times, most recently from 3168d6f to fd72f09 Compare June 25, 2026 09:00
@mfocko

mfocko commented Jun 25, 2026

Copy link
Copy Markdown
Member Author

/packit test

@mfocko
mfocko force-pushed the ci/enable-packit branch from fd72f09 to 3f4850a Compare June 28, 2026 21:33
@mfocko

mfocko commented Jun 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author
image

So that's where the mocks and secrets went…

@mfocko
mfocko force-pushed the ci/enable-packit branch 7 times, most recently from 3c3c9ec to ef9ef34 Compare June 30, 2026 15:13
@mfocko

mfocko commented Jun 30, 2026

Copy link
Copy Markdown
Member Author

/packit test

@mfocko
mfocko force-pushed the ci/enable-packit branch 12 times, most recently from 380ee68 to 595cbc3 Compare July 1, 2026 22:38
@mfocko
mfocko force-pushed the ci/enable-packit branch 3 times, most recently from 9168d5a to a093c17 Compare July 6, 2026 19:36
@mfocko

mfocko commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/packit test

@mfocko
mfocko force-pushed the ci/enable-packit branch 2 times, most recently from 77a84b1 to 8158442 Compare July 8, 2026 21:26
lbarcziova
lbarcziova previously approved these changes Jul 9, 2026
View reviewed changes

@lbarcziova lbarcziova left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great, thanks a lot!

Comment thread .testing-farm.yaml
mfocko added 11 commits July 10, 2026 10:24
@mfocko
ci(packit): add GITLAB_CI_TOKEN for main and mfocko
8f925bd 
Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
ci: switch from CentOS to Fedora
ad92011 
After trying to run the pipeline, I've noticed that CentOS is missing
podman-compose. After a brief chat with @TomasKorbar, it should be OK to
run on the latest Fedora.

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
ci: remove tags
0a84a4a 
Tags have to be KV pairs.

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
fix(tmt): adjust to the changed directory structure
cca93e8 
Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
ci(tmt): move the prepped files to TMT_PLAN_DATA
ebc567d 
Looking at the docs, it appears that the only files that are guaranteed
to be persistent are saved within the $TMT_PLAN_DATA directory, therefore
move the prepped data there and move them back before the test starts.

Excerpt from the docs:

    If you want to use the prepare step to generate data files needed for
    testing during the execute step, move or copy them into
    ${TMT_PLAN_DATA} directory. Only files in this directory are
    guaranteed to be preserved.

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
ci(tmt): add tracing config and .env to the plan
33b7007 
These two are missing from the initial proposed tmt plan.

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
test: wrap test cases in ‹pytest.param()›
4dbdf31 
Wrap the test cases, so that it is easier to see which Jira tickets have
failed the tests, instead of generic `test_case_‹index›`.

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
test: scrape traces from the tests
99ede36 
It is possible to propagate artifacts produced by the tests to the
results of the tests.

Tracing server needs to be running for running the tests, might as well
collect the produced traces from it.

It can be also helpful during the debugging as the produced logs may be
interleaved (test cases are run asynchronously).

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
test: disable agent.trajectory logger
8c2f7ff 
Reasons:
1. quite spammy, also disabled in Sentry
2. produces basically the same output as tracing server (output of which
   is collected and nicely rendered in the HTML)
3. makes the reading of logs from tests impossible; generates around
   ~120MB of logs which makes the UX of viewing it on TF unbearable and
   also download takes time

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
test: report to tmt as custom results
b15791d 
Reasoning:
1. Tests are run concurrently ~> no reasonable logs from pytest
2. We scrape tracing server for debugging purposes ~> additional artifact
3. Each test case is quite complex as it runs an LLM workflow, producing
   traces and also some metrics are collected ~> there's a lot of
   attributes that can be associated with each test

Benefits:
1. Each test case is reported as separate subtest on tmt' side, therefore
   it is easy to bundle all the artifacts, collected metrics and test
   result together.
2. It is possible to tell at first sight which test cases failed, as they
   are reported by Testing Farm view separately and each test case is
   identified by the Jira ticket.
3. Collected metrics (displayed in terminal via pytest as a table) are
   presented as notes:
   - agent
   - tool calls
   - prompt tokens
   - completion tokens
4. Since test cases are run before the actual assertions happen, we can
   also propagate the duration of tests this way. pytest assertions do
   not capture the actual duration of the test case.

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
ci(tmt): split the preparation steps
1fb2752 
They are basically two separate steps, in case one of them fails, it is
easier to trace and debug.

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
mfocko force-pushed the ci/enable-packit branch from 8158442 to 2c9c352 Compare July 10, 2026 08:32
mfocko added 2 commits July 10, 2026 10:34
@mfocko
docs: adjust and move docs for the CI setup
a3a364c 
Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
ci(packit): run on main and prefixed branches
7194556 
I don’t really like the idea of running for »all« branches, as it would
not be unusual to have some automation in repo. And even though the CI
is triggered manually, there is still a read-only token for internal
GitLab present…

Therefore run on ‹main› and any branch that starts with the prefix of
‹test-pr/›. This also groups the refs for testing under ‹test-pr›, so
the cleanup can be done easily from the shell.

Signed-off-by: Matej Focko <mfocko@packit.dev>
@mfocko
mfocko force-pushed the ci/enable-packit branch from 2c9c352 to 7194556 Compare July 10, 2026 08:36
@mfocko
mfocko requested a review from lbarcziova July 10, 2026 08:36
@mfocko

mfocko commented Jul 10, 2026

Copy link
Copy Markdown
Member Author

@lbarcziova Summary of the changes since last review:

  • added expiration to tokens in 8f925bd
  • docs adjusted in a3a364c
  • adjusted trigger to run on main or any in-repo branch with test-pr prefix in 7194556

lbarcziova
lbarcziova approved these changes Jul 10, 2026
View reviewed changes

@lbarcziova lbarcziova left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 let's roll it

@mfocko
mfocko merged commit 2ecef74 into packit:main Jul 10, 2026
11 checks passed
@mfocko
mfocko deleted the ci/enable-packit branch July 10, 2026 08:44
@qodo-for-packit qodo-for-packit Bot mentioned this pull request Sep 2, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lbarcziova lbarcziova lbarcziova approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@mfocko mfocko

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mfocko @lbarcziova