[utsab345]

Fixes #3354.

Adds Pydantic models for the structured output schemas declared in the prompt templates. Conditional fields are optional, and prompt names that are not valid Python identifiers use aliases.

Adds complete validation fixtures and a drift test that checks prompt fields remain represented by the models.

Tests: 337 related unit tests passed. Ruff and pre-commit passed.

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Add typed Pydantic models for prompt outputs

✨ Enhancement 🧪 Tests 🕐 20-40 Minutes

AI Description

• Add typed Pydantic contracts for structured outputs declared across prompt templates.
• Preserve conditional and non-identifier fields through optional types and aliases.
• Validate complete payloads and detect prompt-to-model field drift.

Diagram

graph TD
    P["Prompt Templates"] --> D["Drift Test"] --> M["Output Models"] --> N["Nested Schemas"]
    F["Validation Fixtures"] --> M --> A["Aliases and Optionals"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Generate prompt schemas from canonical models

• ➕ Creates a single source of truth for fields and types
• ➕ Eliminates manual synchronization between TOML declarations and Python models
• ➕ Can expose full JSON Schema constraints to structured-output providers
• ➖ Requires changing prompt rendering and potentially generated prompt text
• ➖ May reduce control over the concise schema syntax presented to language models
• ➖ Introduces broader runtime integration beyond this PR's scope

Recommendation: The PR's mirrored-model approach is the best low-risk incremental solution because it adds typed contracts without changing existing prompts or runtime behavior, while the drift test protects field coverage. If these models become runtime contracts, generating prompt schema declarations from them should be considered to eliminate type and optionality drift that the current field-only check cannot detect.

Files changed (2) +314 / -0

Enhancement (1) +170 / -0

output_models.pyDefine typed contracts for structured prompt outputs +170/-0

Define typed contracts for structured prompt outputs

• Adds Pydantic models for review, suggestion, description, labeling, ranking, and documentation prompt outputs. Optional fields represent conditional prompt sections, while aliases and lowercase exports preserve prompt-specific names.

pr_agent/algo/output_models.py

Tests (1) +144 / -0

test_output_models.pyValidate output contracts and guard against prompt drift +144/-0

Validate output contracts and guard against prompt drift

• Adds parameterized validation fixtures covering the new output models and verifies aliased prompt keys. A drift test scans configured prompt declarations and ensures every declared field remains represented by its corresponding model.

tests/unittest/test_output_models.py

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 📜 Skill insights (0)

1. Review effort example is rejected ✓ Resolved 📎 Requirement gap ≡ Correctness

Description

Review.estimated_effort_to_review requires a StrictInt, but the duplicate reviewer example still
emits 3 through a YAML | block scalar, which parses as a string. When
duplicate_prompt_examples and effort estimation are enabled, that documented payload disagrees
with the new model while the primary example and fixture use an integer.

Code

pr_agent/algo/output_models.py[R56-58]

+    estimated_effort_to_review: Optional[StrictInt] = Field(
+        default=None, alias="estimated_effort_to_review_[1-5]", ge=1, le=5
+    )

Evidence

Compliance rule 10 requires the reviewer schema and examples to use consistent value types. The new
model accepts only a strict integer, while the duplicate prompt example uses a YAML block scalar
that loads as a string.

Correct Known Reviewer Prompt Schema Drift
pr_agent/algo/output_models.py[56-58]
pr_agent/settings/pr_reviewer_prompts.toml[303-305]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The duplicate reviewer example represents `estimated_effort_to_review_[1-5]` as a YAML block scalar, producing a string that the new strict integer model rejects.
## Fix Focus Areas
- pr_agent/settings/pr_reviewer_prompts.toml[303-305]
## Recommended Fix
Change the duplicate example to `estimated_effort_to_review_[1-5]: 3`, matching the primary example, fixture, prompt schema, and strict integer model.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Documentation output goes unvalidated ✓ Resolved 📎 Requirement gap ≡ Correctness

Description

output_models.py, the fixture parametrization, and PROMPT_MODELS omit the Code Documentation
schema declared by pr_add_docs.toml. The add-documentation tool therefore has no aliased
executable model or complete fixture, and edits to its four output fields never reach the new drift
test.

Code

tests/unittest/test_output_models.py[R261-264]

+    "code_suggestions/pr_code_suggestions_prompts.toml": {"CodeSuggestion": CodeSuggestion, "PRCodeSuggestions": PRCodeSuggestions},
+    "code_suggestions/pr_code_suggestions_prompts_not_decoupled.toml": {"CodeSuggestion": CodeSuggestion, "PRCodeSuggestions": PRCodeSuggestions},
+    "code_suggestions/pr_code_suggestions_reflect_prompts.toml": {"CodeSuggestionFeedback": CodeSuggestionFeedback, "PRCodeSuggestionsFeedback": PRCodeSuggestionsFeedback},
+}

Evidence

The add-documentation prompt declares a structured output with space-containing field names, while
the new output-model module ends without representing it and the added registry omits its prompt
file. Consequently, the parametrized fixtures and drift checks do not cover this tool, violating the
requirements for complete models, aliases, fixtures, fixture validation, and schema-drift detection.

Implement real Pydantic output models
Support aliases and optional require fields
Provide one complete fixture per tool
Validate all fixtures with parametrized tests
Detect TOML and model schema drift
pr_agent/settings/pr_add_docs.toml[27-52]
pr_agent/algo/output_models.py[175-182]
tests/unittest/test_output_models.py[248-264]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The structured output from `pr_add_docs.toml` is absent from the new Pydantic models, complete fixtures, and schema-drift coverage.
## Fix Focus Areas
- pr_agent/algo/output_models.py[180-182]
- tests/unittest/test_output_models.py[69-113]
- tests/unittest/test_output_models.py[248-264]
## Recommended Fix
Add models for the top-level `Code Documentation` list and each documentation item, using aliases for names containing spaces and a literal for placement. Add a concrete fixture for the add-documentation tool and extend the drift test or add a schema-specific test that compares all four TOML fields with the model.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. New response model keeps a typo ✓ Resolved 📘 Rule violation ⚙ Maintainability

Description

PRRankRespones misspells Responses, and the same misspelling is copied into the new tests and
prompt-model registry. A later maintainer searching for the correctly named response model can miss
this contract or propagate the typo into additional schema tooling.

Code

pr_agent/algo/output_models.py[139]

+class PRRankRespones(BaseModel):

Evidence

Compliance rule 1 requires new classes to follow established naming conventions. The added model
declares the misspelled name, and the new tests propagate that spelling through imports, fixtures,
and the prompt registry.

Rule 1: Consistent Naming Conventions
pr_agent/algo/output_models.py[139-143]
tests/unittest/test_output_models.py[24-24]
tests/unittest/test_output_models.py[217-217]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new response model is named `PRRankRespones`, preserving a spelling error and spreading it through the tests and prompt-model registry.
## Fix Focus Areas
- pr_agent/algo/output_models.py[139-143]
- tests/unittest/test_output_models.py[24-24]
- tests/unittest/test_output_models.py[107-107]
- tests/unittest/test_output_models.py[217-217]
- pr_agent/settings/pr_evaluate_prompt_response.toml[47-49]
## Recommended Fix
Rename the model and prompt declaration to `PRRankResponses`, then update every import, fixture, registry key, and validation reference consistently.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View high (6)

4. Reviewer examples teach obsolete output ✓ Resolved 📎 Requirement gap ≡ Correctness

Description

The new TicketCompliance contract requires requires_further_human_verification, but its
associated reviewer examples still emit overall_compliance_level instead. Whenever related-ticket
output is enabled, both examples demonstrate a payload that omits the modeled field and includes the
dropped one.

Code

pr_agent/algo/output_models.py[33]

+    requires_further_human_verification: str

Evidence

Compliance rule 11 explicitly requires removing overall_compliance_level from the reviewer
examples. The new model and prompt schema require requires_further_human_verification, while both
examples on the PR branch continue to show the obsolete field.

Correct known reviewer prompt schema and example mismatches
pr_agent/algo/output_models.py[28-33]
pr_agent/settings/pr_reviewer_prompts.toml[79-84]
pr_agent/settings/pr_reviewer_prompts.toml[142-153]
pr_agent/settings/pr_reviewer_prompts.toml[293-304]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new ticket-compliance model requires `requires_further_human_verification`, while both related-ticket examples still use the obsolete `overall_compliance_level` field.
## Fix Focus Areas
- pr_agent/settings/pr_reviewer_prompts.toml[142-153]
- pr_agent/settings/pr_reviewer_prompts.toml[293-304]
## Recommended Fix
Replace `overall_compliance_level` with `requires_further_human_verification` in both reviewer examples, preserving their existing YAML block formatting and Jinja conditions.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Suggestions can omit their labels ✓ Resolved 📎 Requirement gap ≡ Correctness

Description

CodeSuggestion.label is declared as Optional[str] = None, although both conditional prompt
branches declare it as a required str with no default. A payload that omits the label or supplies
null therefore validates through PRCodeSuggestions, even though no rendered version of the prompt
permits that shape.

Code

pr_agent/algo/output_models.py[73]

+    label: Optional[str] = None

Evidence

Both branches of the conditional code-suggestion prompt use different label descriptions but declare
the same required label: str field, whereas the Python model defines label as `Optional[str] =
None`. The model therefore accepts both omission and null instead of representing the prompt schemas
as required by Rule 6.

Implement real Pydantic output models
pr_agent/algo/output_models.py[66-74]
pr_agent/settings/code_suggestions/pr_code_suggestions_prompts.toml[66-77]
pr_agent/algo/output_models.py[66-73]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`CodeSuggestion.label` is optional even though every rendered prompt branch requires the field, so payloads that omit it or set it to null incorrectly pass validation.
## Fix Focus Areas
- pr_agent/algo/output_models.py[66-74]
- pr_agent/settings/code_suggestions/pr_code_suggestions_prompts.toml[66-77]
## Recommended Fix
Declare `label` as a required `str` with no default. Retain fixture coverage showing that prompt-shaped suggestions validate, and add a negative test proving that payloads without `label` fail validation.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

6. Reviewer score types stay inconsistent ✓ Resolved 📎 Requirement gap ≡ Correctness

Description

Review.score is modeled as Optional[int] and represented by the fixture as the integer 89,
while the reviewer prompt declares score: str. When score output is enabled, schema generation and
typed consumers use an integer contract that does not preserve the prompt-declared type, and the new
drift test fails to detect the disagreement.

Code

pr_agent/algo/output_models.py[53]

+    score: Optional[int] = None

Evidence

The reviewer prompt explicitly declares score: str, whereas the newly added Python model declares
Optional[int] and the fixture supplies a bare integer. This fixture validates only the model’s
integer representation rather than proving parity with the prompt, despite Rule 11 requiring known
reviewer type discrepancies to be aligned.

Correct known reviewer prompt inconsistencies
pr_agent/settings/pr_reviewer_prompts.toml[114-116]
pr_agent/algo/output_models.py[53-53]
tests/unittest/test_output_models.py[46-46]
tests/unittest/test_output_models.py[41-49]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The reviewer prompt, output model, and fixture disagree about whether `score` is a string or integer, and the drift validation does not catch the mismatch.
## Fix Focus Areas
- pr_agent/algo/output_models.py[53-53]
- pr_agent/settings/pr_reviewer_prompts.toml[114-116]
- tests/unittest/test_output_models.py[46-46]
## Recommended Fix
Choose the intended reviewer score type and apply it consistently to the prompt declaration, output model, examples, and complete fixture. Update the drift validation so a future mismatch between the prompt and typed model is detected.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

7. Schema gaps pass fixture validation ✓ Resolved 📎 Requirement gap ≡ Correctness

Description

The PRDescription, PRDescriptionHeaders, and file-walkthrough fixtures omit modeled conditional
fields such as description, changes_diagram, pr_files, and changes_summary. When those
prompt settings are enabled, the parametrized test still passes without validating a concrete value
for the complete generated output structure.

Code

tests/unittest/test_output_models.py[R89-92]

+        (PRDescription, {"type": ["Bug fix"], "description": "Fix a runtime failure.", "title": "Fix runtime failure"}),
+        (PRDescriptionHeaders, {"type": ["Tests"], "title": "Describe the test changes"}),
+        (FileDescription, {"filename": "src/app.py", "changes_title": "Handle runtime failures", "label": "bug fix"}),
+        (PRFilesWalkthrough, {"pr_files": [{

Evidence

Rule 8 requires one complete fixture per tool. The added description fixtures omit fields declared
by their prompts and represented by the new models, so successful validation does not establish
completeness.

Provide one complete fixture per tool
tests/unittest/test_output_models.py[89-100]
pr_agent/algo/output_models.py[101-124]
pr_agent/settings/pr_description_prompts.toml[52-71]
pr_agent/settings/pr_description_only_description_prompts.toml[49-56]
pr_agent/settings/pr_description_only_files_prompts.toml[35-44]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Several fixtures omit conditional fields and therefore do not exercise each tool's complete modeled output structure.
## Fix Focus Areas
- tests/unittest/test_output_models.py[89-100]
## Recommended Fix
Populate every modeled field in each applicable tool fixture with concrete values, including descriptions, diagrams, file lists, and file change summaries, while keeping all fixtures in the parametrized validation test.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

8. Configured labels fail validation ✓ Resolved 🐞 Bug ≡ Correctness

Description

Labels.labels is fixed to the five-member Label enum even though set_custom_labels renders an
enum from arbitrary repository configuration. When custom labels are enabled, outputs valid for the
rendered prompt cannot be validated by this model unless their values happen to match the built-in
set.

Code

pr_agent/algo/output_models.py[R135-136]

+class Labels(BaseModel):
+    labels: List[Label]

Evidence

The prompt substitutes a dynamically generated label class when custom labels are enabled, and
set_custom_labels builds its enum members from arbitrary configured labels. The added model
instead always restricts validation to five hard-coded values.

pr_agent/settings/pr_custom_labels.toml[17-33]
pr_agent/algo/utils.py[1295-1319]
pr_agent/algo/output_models.py[127-136]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The static `Label` enum rejects values produced by prompts rendered from repository-specific custom-label configuration.
## Fix Focus Areas
- pr_agent/algo/output_models.py[127-136]
## Recommended Fix
Represent `Labels.labels` using strings or construct the enum dynamically from the active custom-label configuration, while preserving validation of the built-in configuration where applicable.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

9. Prompt schema drift stays undetected ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

test_prompt_fields_are_present_in_output_models extracts only field names and performs a one-way
subset comparison. Type, enum, constraint, and removed-field changes therefore pass, as demonstrated
by the existing score string-versus-integer mismatch and the unchecked prompt enum declarations.

Code

tests/unittest/test_output_models.py[R141-144]

+            declared = set(re.findall(r"^    ([A-Za-z_][A-Za-z0-9_\[\]-]*):", block, re.MULTILINE))
+            model_fields = set(model.model_fields)
+            aliases = {field.alias for field in model.model_fields.values() if field.alias}
+            assert declared <= model_fields | aliases, f"{relative_path}: {class_name} has unmodelled fields"

Evidence

The regular expression captures only the identifier before each annotation, and the subset assertion
permits model-only stale fields. The mapping also excludes PRType and Label, while the current
score mismatch proves that incompatible annotations already pass this test.

tests/unittest/test_output_models.py[111-126]
tests/unittest/test_output_models.py[130-144]
pr_agent/settings/pr_reviewer_prompts.toml[114-116]
pr_agent/algo/output_models.py[53-53]
pr_agent/settings/pr_description_prompts.toml[37-42]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The drift test checks only that current prompt field names exist in the models, leaving most schema changes invisible.
## Fix Focus Areas
- tests/unittest/test_output_models.py[111-144]
## Recommended Fix
Parse and compare annotations, aliases, requiredness, enum members, and Pydantic constraints in addition to names. Compare the appropriate union of conditional and shared prompt variants so deleting a field from every relevant prompt also fails without treating intentionally absent variant fields as drift.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

10. Duplicate docs pass validation ✓ Resolved 🐞 Bug ≡ Correctness

Description

CodeDocumentation.code_documentation uses a plain List[...] and has no validator for the prompt
schema's uniqueItems: true constraint. A response containing the same documentation item more than
once therefore validates, and the singleton fixture cannot expose this schema drift.

Code

pr_agent/algo/output_models.py[168]

+    code_documentation: List[CodeDocumentationItem] = Field(alias="Code Documentation")

Evidence

The add-docs schema explicitly marks the Code Documentation array as unique, while the executable
model declares an unconstrained list and the test validates only a one-item payload.

pr_agent/settings/pr_add_docs.toml[27-33]
pr_agent/algo/output_models.py[165-168]
tests/unittest/test_output_models.py[305-315]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`CodeDocumentation.code_documentation` does not enforce the add-docs prompt's `uniqueItems: true` constraint, allowing duplicate documentation items to validate.
## Fix Focus Areas
- pr_agent/algo/output_models.py[165-168]
- tests/unittest/test_output_models.py[305-315]
## Recommended Fix
Add validation that rejects duplicate documentation items based on their complete serialized field values, and add a negative test containing two identical items.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

11. The ranking example fails validation ✓ Resolved 🐞 Bug ≡ Correctness

Description

PRRankResponses.which_response_was_better accepts only integer literals 0, 1, and 2, but the
adjacent prompt example still supplies the quoted value "X". When a response mirrors the example,
model validation fails before the ranking can be represented, while the fixture avoids the
inconsistency by supplying 1.

Code

pr_agent/algo/output_models.py[R171-173]

+class PRRankResponses(BaseModel):
+    which_response_was_better: Literal[0, 1, 2]
+    why: str

Evidence

The model restricts the field to numeric literals, whereas the prompt example shows a quoted
nonnumeric value; the new fixture validates a different numeric payload instead.

pr_agent/algo/output_models.py[171-175]
pr_agent/settings/pr_evaluate_prompt_response.toml[49-60]
tests/unittest/test_output_models.py[113-113]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The response-ranking prompt example emits `"X"`, which cannot validate against the new integer-literal output model.
## Fix Focus Areas
- pr_agent/settings/pr_evaluate_prompt_response.toml[57-62]
- tests/unittest/test_output_models.py[113-113]
## Recommended Fix
Replace `"X"` in the example with one valid numeric ranking such as `1`, keeping it unquoted, and add a test that parses and validates the concrete example shape.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

12. Documentation examples fail validation ✓ Resolved 🐞 Bug ≡ Correctness

Description

CodeDocumentationItem aliases only relevant line and forbids extra keys, while the prompt's
concrete example emits relevant lines. An output following that example is rejected by the new
model and also misses the singular key consumed by push_inline_docs, so the fixture validates only
the schema spelling rather than a representative prompt response.

Code

pr_agent/algo/output_models.py[R159-160]

+    relevant_file: str = Field(alias="relevant file")
+    relevant_line: StrictInt = Field(alias="relevant line", ge=1)

Evidence

The new model accepts the singular alias and forbids unknown keys, while the prompt example uses the
plural form. The production consumer also directly indexes the singular key, and the added test
constructs its own singular payload instead of covering the contradictory example.

pr_agent/algo/output_models.py[156-162]
pr_agent/settings/pr_add_docs.toml[54-65]
pr_agent/tools/pr_add_docs.py[115-122]
tests/unittest/test_output_models.py[305-315]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The add-documentation prompt example emits `relevant lines`, but the new model and production consumer require `relevant line`. Outputs following the example therefore fail typed validation and document insertion.
## Fix Focus Areas
- pr_agent/settings/pr_add_docs.toml[54-60]
- tests/unittest/test_output_models.py[305-315]
## Recommended Fix
Change the example key from `relevant lines` to `relevant line`, then add a test that extracts or reproduces the complete example shape and validates it with `CodeDocumentation`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View medium (9)

13. The reviewer example fails validation 🐞 Bug ≡ Correctness

Description

relevant_tests: No is now a plain YAML scalar, so load_yaml resolves it to boolean False while
Review.relevant_tests accepts only the string literals and its pre-validator only strips strings.
Any reviewer response that follows this changed example reaches typed validation as the wrong type,
including the example the new contract is intended to cover.

Code

pr_agent/settings/pr_reviewer_prompts.toml[172]

+  relevant_tests: No

Evidence

The changed prompt emits bare No, while the model permits only the strings Yes and No and
normalizes only string inputs. Production review parsing passes model output through
yaml.safe_load, whose YAML scalar resolution converts bare No to False, creating the
demonstrated type mismatch.

pr_agent/settings/pr_reviewer_prompts.toml[169-172]
pr_agent/algo/output_models.py[64-74]
pr_agent/algo/utils.py[1023-1032]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The reviewer example emits `No` as an unquoted YAML scalar, which `yaml.safe_load` converts to boolean `False`; this conflicts with the model's required string literal `"No"`.
## Fix Focus Areas
- pr_agent/settings/pr_reviewer_prompts.toml[172-172]
- tests/unittest/test_output_models.py[115-116]
## Recommended Fix
Quote the example value as `relevant_tests: "No"` so YAML parsing preserves it as a string. Add a regression test that parses this representative YAML through `load_yaml` before validating it with `PRReview`, ensuring fixtures exercise the actual parsing boundary.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

14. Reviewer decisions accept unknown values ✓ Resolved 🐞 Bug ≡ Correctness

Description

Review.risk_level and Review.merge_recommendation use unrestricted strings instead of the finite
choices specified by their prompts. Payloads such as critical or approve therefore validate when
risk assessment or merge recommendation output is enabled, despite not representing any supported
reviewer decision.

Code

pr_agent/algo/output_models.py[R49-50]

+    risk_level: Optional[str] = None
+    merge_recommendation: Optional[str] = None

Evidence

The model fields accept any string, while the prompt explicitly requires one of three enumerated
values for each field.

pr_agent/algo/output_models.py[49-50]
pr_agent/settings/pr_reviewer_prompts.toml[103-106]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Reviewer risk levels and merge recommendations currently validate arbitrary strings even though their prompts define finite sets of supported values.
## Fix Focus Areas
- pr_agent/algo/output_models.py[49-50]
- pr_agent/settings/pr_reviewer_prompts.toml[103-106]
- tests/unittest/test_output_models.py[122-152]
## Recommended Fix
Type `risk_level` as an optional literal containing `low`, `medium`, and `high`, and type `merge_recommendation` as an optional literal containing `safe_to_merge`, `merge_with_caution`, and `changes_required`. Add validation tests showing that values outside those sets are rejected.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

15. Test coverage answers accept ambiguity ✓ Resolved 🐞 Bug ≡ Correctness

Description

Review.relevant_tests is an unrestricted optional string even though the corresponding prompt asks
a yes/no question. When test reporting is enabled, values such as maybe or arbitrary prose pass
validation and no longer provide the binary result consumers expect.

Code

pr_agent/algo/output_models.py[54]

+    relevant_tests: Optional[str] = None

Evidence

The output model permits every string, whereas the prompt defines this field as the answer to a
yes/no question.

pr_agent/algo/output_models.py[53-55]
pr_agent/settings/pr_reviewer_prompts.toml[117-119]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The reviewer output model accepts arbitrary strings for a prompt field whose valid answers are yes and no.
## Fix Focus Areas
- pr_agent/algo/output_models.py[54-54]
- pr_agent/settings/pr_reviewer_prompts.toml[117-119]
- tests/unittest/test_output_models.py[122-152]
## Recommended Fix
Change `relevant_tests` to an optional literal containing the exact accepted yes/no spellings used by the prompt, then add a validation test proving an unsupported answer is rejected.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

16. Pull request types can be empty ✓ Resolved 🐞 Bug ≡ Correctness

Description

PRDescription.type and PRDescriptionHeaders.type use unconstrained lists, allowing an empty
collection to validate. Both description prompts require one or more types, so this affects both the
full-description and headers-only output variants.

Code

pr_agent/algo/output_models.py[109]

+    type: List[PRType]

Evidence

Neither model imposes a minimum length, while both prompt declarations explicitly say the output
must contain one or more types.

pr_agent/algo/output_models.py[108-117]
pr_agent/settings/pr_description_prompts.toml[61-62]
pr_agent/settings/pr_description_only_description_prompts.toml[49-50]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Both pull request description models accept an empty type list despite their prompts requiring one or more types.
## Fix Focus Areas
- pr_agent/algo/output_models.py[108-117]
- pr_agent/settings/pr_description_prompts.toml[61-62]
- pr_agent/settings/pr_description_only_description_prompts.toml[49-50]
- tests/unittest/test_output_models.py[122-152]
## Recommended Fix
Apply a minimum list length of one to the `type` fields in both description models and add rejection tests for empty lists in each output variant.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

17. Review validation accepts invalid scores ✓ Resolved 🐞 Bug ≡ Correctness

Description

Review.estimated_effort_to_review and Review.score are unconstrained integers despite their
declared 1–5 and 0–100 scales. Values such as effort 0 or score 101 therefore validate whenever
those conditional outputs are enabled.

Code

pr_agent/algo/output_models.py[R46-48]

+    estimated_effort_to_review: Optional[int] = Field(
+        default=None, alias="estimated_effort_to_review_[1-5]"
+    )

Evidence

The model uses plain optional integers, while the prompt explicitly defines inclusive scales for
both fields; no Pydantic bounds prevent values outside those scales.

pr_agent/algo/output_models.py[46-53]
pr_agent/settings/pr_reviewer_prompts.toml[99-100]
pr_agent/settings/pr_reviewer_prompts.toml[114-115]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The typed review model accepts effort and quality scores outside the ranges specified by the reviewer prompt.
## Fix Focus Areas
- pr_agent/algo/output_models.py[46-53]
- tests/unittest/test_output_models.py[122-137]
## Recommended Fix
Add inclusive Pydantic bounds of 1–5 to `estimated_effort_to_review` and 0–100 to `score`, preserving their optional defaults and the effort alias. Add negative validation tests for values below and above both ranges.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

18. Evaluation accepts impossible scores ✓ Resolved 🐞 Bug ≡ Correctness

Description

CodeSuggestionFeedback.suggestion_score, PRRankRespones.score_response1, and score_response2
are modeled as unrestricted integers. Suggestion scores outside 0–10 and response scores outside
1–10 consequently pass validation for the evaluation outputs.

Code

pr_agent/algo/output_models.py[85]

+    suggestion_score: int

Evidence

All three model fields are plain integers, but the reflection prompt limits suggestion scores to
0–10 and the response-evaluation prompt limits both response scores to 1–10.

pr_agent/algo/output_models.py[80-86]
pr_agent/algo/output_models.py[139-143]
pr_agent/settings/code_suggestions/pr_code_suggestions_reflect_prompts.toml[51-57]
pr_agent/settings/pr_evaluate_prompt_response.toml[49-53]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The typed evaluation models accept scores outside the ranges required by their respective prompt contracts.
## Fix Focus Areas
- pr_agent/algo/output_models.py[80-86]
- pr_agent/algo/output_models.py[139-143]
- tests/unittest/test_output_models.py[122-137]
## Recommended Fix
Apply inclusive Pydantic bounds of 0–10 to `suggestion_score` and 1–10 to both response score fields. Add validation tests covering values immediately below and above each allowed range.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

19. Docs validation accepts invalid selectors ✓ Resolved 🐞 Bug ≡ Correctness

Description

DocHelper.question_is_relevant and FileIdxAndPath.idx are unrestricted integers rather than
enforcing their documented domains. A relevance value other than 0 or 1, or a negative document
index, therefore validates even though the corresponding documentation prompts prohibit it.

Code

pr_agent/algo/output_models.py[155]

+    question_is_relevant: Optional[int] = None

Evidence

The models apply no numeric constraints, while one prompt defines the relevance flag as 0 or 1 and
the headings prompt explicitly prohibits negative indices.

pr_agent/algo/output_models.py[151-160]
pr_agent/settings/pr_help_docs_prompts.toml[17-21]
pr_agent/settings/pr_help_docs_headings_prompts.toml[43-49]
pr_agent/settings/pr_help_docs_headings_prompts.toml[94-96]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The documentation output models accept relevance flags and file indices outside the values permitted by their prompts.
## Fix Focus Areas
- pr_agent/algo/output_models.py[151-160]
- tests/unittest/test_output_models.py[122-137]
## Recommended Fix
Constrain `question_is_relevant` to inclusive values 0–1 while retaining its optional default for the prompt variant that omits it, and constrain `idx` to non-negative integers. Add rejection tests for relevance values outside 0–1 and for negative indices.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

20. Invalid response rankings still pass ✓ Resolved 🐞 Bug ≡ Correctness

Description

PRRankRespones.which_response_was_better uses an unrestricted int instead of the prompt's
three-value literal type. Payloads containing values outside 0, 1, and 2 therefore pass model
validation despite violating the declared output contract.

Code

pr_agent/algo/output_models.py[R139-143]

+class PRRankRespones(BaseModel):
+    which_response_was_better: int
+    why: str
+    score_response1: int
+    score_response2: int

Evidence

The prompt annotation restricts the choice to exactly three integers, while the added model places
no value restriction on that field. The positive fixture exercises only one allowed value and cannot
expose the missing constraint.

pr_agent/settings/pr_evaluate_prompt_response.toml[47-53]
pr_agent/algo/output_models.py[139-143]
tests/unittest/test_output_models.py[95-96]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The ranking model drops the prompt's literal constraint and accepts unsupported response choices.
## Fix Focus Areas
- pr_agent/algo/output_models.py[139-143]
- pr_agent/settings/pr_evaluate_prompt_response.toml[49-53]
## Recommended Fix
Import `Literal`, type `which_response_was_better` as `Literal[0, 1, 2]`, and add rejection tests for unsupported values.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

21. Oversized prompt lists still pass ✓ Resolved 🐞 Bug ≡ Correctness

Description

Review.can_be_split and PRDescription.pr_files omit the maximum lengths declared by their prompt
fields. More than three proposed sub-pull requests or more than twenty file descriptions
consequently validate against the typed models.

Code

pr_agent/algo/output_models.py[59]

+    can_be_split: Optional[List[SubPR]] = None

Evidence

The reviewer prompt sets max_items=3 and the description prompt sets max_items=20, but both
added models use unconstrained lists. Their fixtures contain only one item, so the missing upper
bounds remain untested.

pr_agent/settings/pr_reviewer_prompts.toml[130-132]
pr_agent/settings/pr_description_prompts.toml[70-72]
pr_agent/algo/output_models.py[59-59]
pr_agent/algo/output_models.py[108-113]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Two list fields lose maximum-size constraints that are part of their prompt schemas.
## Fix Focus Areas
- pr_agent/algo/output_models.py[59-59]
- pr_agent/algo/output_models.py[108-113]
- pr_agent/settings/pr_reviewer_prompts.toml[130-132]
- pr_agent/settings/pr_description_prompts.toml[70-72]
## Recommended Fix
Apply Pydantic list constraints of three items to `can_be_split` and twenty items to `pr_files`, then add boundary and over-limit validation tests.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

• Author self-review: I have reviewed the code review findings, and addressed the relevant ones.

Tip of the day

💡 Did you know, you can ask Qodo to dismiss a finding you disagree with, with your reason on record

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit 03f2606

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit c12aa5e

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit a9ef8c7

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit 681306a

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit 2701800

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit 9cf870b

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit 6aa533d

[utsab345]

@IsmaelMartinez all requested fixes are done and pushed. The output models, prompt contracts, and tests are updated. Validation is green.

[IsmaelMartinez]

Thanks for turning this round in two hours, and for taking every Qodo thread to a commit.

One ask, inline: the system example was switched to a bare No, which PyYAML reads as the boolean False, so the shape the prompt now teaches fails the model's own Literal["Yes","No"]. The block scalar the user example still uses is the safe form.

Everything else held: fixtures validate, the guard goes red on a renamed or retyped field, nested classes included, and it merges cleanly with #3366.

Worth a follow-up, not a blocker: the guard is one-directional and the enum values are pinned by hand rather than parsed.

[IsmaelMartinez]

load_yaml is PyYAML 1.1, so a bare No is the boolean False (load_yaml('review:\n relevant_tests: No') gives False), and the example fails Literal["Yes","No"] under every require_* toggle. The block scalar validates everywhere; applied, 30 passed.

Suggested change

	relevant_tests: No
	relevant_tests: |
	No

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit 0d522fc

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit c9b5627

[utsab345]

Done. All requested collaborator and review fixes are applied, and the checks are passing.

[IsmaelMartinez]

Thanks for switching the example back to the block scalar within five minutes, and for picking up Qodo's duplicate-example point yourself in the next push rather than waiting on #3366; approving.

On today's main, with #3366 in, the branch merges clean and the two agree on every example line. The suite is main plus your tests, the drift guard goes red with the prompt files restored, and CI is green. The evaluate example edit is inert for now: that prompt file is not registered in config_loader.py, so nothing loads it.