Skip to content

Enable rebuild+backport MR consolidation #689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
opohorel merged 3 commits into from
Jul 21, 2026
Merged

Enable rebuild+backport MR consolidation #689

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3c8f6a1
Enable rebuild+backport MR consolidation
opohorel Jul 16, 2026
8e5f27f
Use sorting approach for prioritizing of MRs to consolidate
TomasKorbar Jul 21, 2026
7579fcc
Refactor consolidation job submission into shared helper
TomasKorbar Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 38 additions & 3 deletions docs/mr_consolidation_architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,17 @@ originals as consolidated.
┌──────────────┐ submit_merge_job() ┌────────────────────┐
│ │ ─────────────────────────────▶ │ │
│ Backport │ (after filing an MR, if │ Redis Queue │
│ Agent │ ymir.yaml enables it │ (Hash-based, │
│ │ it for the package) │ per pkg/branch) │
│ Agent │ ymir.yaml enables it) │ (Hash-based, │
│ │ │ per pkg/branch) │
└──────────────┘ └────────┬───────────┘
│
┌──────────────┐ submit_merge_job() │
│ Rebuild │ ─────────────────────────────▶ │
│ Agent │ (after filing an MR, if │
│ │ ymir.yaml enables it) │
└──────────────┘ │
│
┌──────────────┐ submit_merge_job() │
│ Jira Issue │ ─────────────────────────────▶ │
│ Fetcher │ (when ymir_consolidate_base │
│ │ + _next labels match) │
Expand All @@ -38,7 +44,7 @@ originals as consolidated.
│ 1. list_open_mrs │
│ 2. fork & clone │
│ 3. filter by HEAD │
│ 4. LLM merges │
│ 4. merge/append │
│ 5. build verify │
│ 6. log & commit │
│ 7. mark originals │
Expand Down Expand Up @@ -295,6 +301,34 @@ exactly those two MRs — it does **not** fall back to the "oldest two" heuristi
`ymir_consolidate_next` for the same package/branch (or vice versa), the fetcher logs
a warning and leaves the labels untouched for the next sweep.

## Rebuild + Backport Consolidation

When a package has both an open backport MR (`ymir_backport`) and a rebuild MR
(`ymir_rebuild`) for the same branch, the consolidation agent can fold the rebuild
into the backport. This avoids a double Release bump since the backport already
bumps Release.

### How it works

1. **Discovery**: `list_open_mrs` searches for both `ymir_backport` and `ymir_rebuild`
labeled MRs. At least one backport MR must be present.
2. **Selection priority**: backport+backport pairs are consolidated first. Only when
fewer than 2 backport MRs exist does the agent pick a backport+rebuild pair.
3. **Append-only flow**: The backport branch is cherry-picked as the base. The rebuild
branch is NOT cherry-picked (its Release bump + changelog are redundant). Instead,
only the rebuild's Jira ticket(s) and CVE(s) are appended to the commit message.
4. **CVE propagation**: CVE IDs from both MRs are collected. The `CVE:` line in the
commit message is updated only if the base backport already had one (respecting
per-package maintainer rules that may omit it).
5. **Output**: The consolidated MR is labeled `ymir_backport`. The original rebuild MR
is labeled `ymir_consolidated`.

### Triggering

Both the backport agent and the rebuild agent submit consolidation jobs after
creating their MRs (when `ymir.yaml` enables consolidation). The consolidation
agent discovers eligible pairs regardless of which agent triggered the job.

## Safety Invariants

| Invariant | Mechanism |
Expand All @@ -321,6 +355,7 @@ a warning and leaves the labels untouched for the next sweep.
| `ymir/common/constants.py` | `MERGE_CONSOLIDATION_QUEUE` Redis key name |
| `ymir/tools/privileged/gitlab.py` | `ListProjectMergeRequestsTool`, `AddMergeRequestLabelsTool` |
| `ymir/agents/backport_agent.py` | `submit_consolidation_job` step (triggers consolidation) |
| `ymir/agents/rebuild_agent.py` | `submit_consolidation_job` step (triggers consolidation for rebuild+backport) |
| `ymir/jira_issue_fetcher/jira_issue_fetcher.py` | Label-triggered consolidation scanning (`_process_consolidation_labels`) |
| `ymir/agents/tests/unit/test_mr_consolidation.py` | Unit tests for Redis queue logic |
| `ymir/agents/tests/e2e/mr_consolidation/` | E2E test suite with LLM judge evaluation |
1 change: 1 addition & 0 deletions jira_label_workflow_routing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ These labels are applied to GitLab merge requests (not Jira issues):
| Label | Purpose | Effect |
|-------|---------|--------|
| `ymir_backport` | Marks an MR as a backport | Used by the consolidation agent to discover candidate MRs |
| `ymir_rebuild` | Marks an MR as a rebuild | Used by the consolidation agent to discover rebuild MRs for backport+rebuild consolidation |
| `ymir_consolidated` | Marks an MR that has been folded into a consolidated MR | The MR stays open but is excluded from future consolidation searches |

## Queue Types Summary
Expand Down
40 changes: 7 additions & 33 deletions ymir/agents/backport_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
from ymir.agents.observability import setup_observability
from ymir.agents.package_update_steps import PackageUpdateState
from ymir.agents.reasoning_agent import ReasoningAgent
from ymir.agents.tasks import InvalidConsolidationConfigError, fetch_consolidation_config, submit_merge_job
from ymir.agents.tasks import InvalidConsolidationConfigError
from ymir.agents.utils import (
check_subprocess,
format_mr_triage_details,
Expand Down Expand Up @@ -729,7 +729,12 @@ async def submit_consolidation_job(state):
return "comment_in_jira"

try:
config = await fetch_consolidation_config(state.package, gateway_tools)
await tasks.try_submit_consolidation_job(
state.package,
state.dist_git_branch,
gateway_tools,
redis_conn,
)
except InvalidConsolidationConfigError as e:
logger.warning("Invalid consolidation config for %s: %s", state.package, e)
await tasks.comment_in_jira(
Expand All @@ -744,37 +749,6 @@ async def submit_consolidation_job(state):
available_tools=gateway_tools,
user_triggered=user_triggered,
)
return "comment_in_jira"

try:
if not config.merge_mrs:
logger.info(
"MR consolidation not enabled for %s, skipping",
state.package,
)
return "comment_in_jira"

if redis_conn is not None:
submitted = await submit_merge_job(
redis_conn,
state.package,
state.dist_git_branch,
release_strategy=config.release_strategy.value,
)
if submitted:
logger.info(
"Submitted consolidation job for %s/%s",
state.package,
state.dist_git_branch,
)
else:
logger.info(
"Consolidation job already queued for %s/%s",
state.package,
state.dist_git_branch,
)
else:
logger.info("No Redis connection (direct mode), skipping consolidation job submission")
except Exception as e:
logger.warning("Failed to submit consolidation job: %s", e)

Expand Down
Loading
Loading