Repository navigation
fix(planning): apply design defaults on every non-trivial design #8423
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
7 changes: 7 additions & 0 deletions
7
.changes/planning/fix-design-defaults-unconditional-b31501b2.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| bump: patch | ||
| --- | ||
|
|
||
| ### Fixed | ||
|
|
||
| - `/planning:design`: a non-trivial design applies `reference/design-defaults.md`, and so writes the Principles thread `/planning:design-handoff` requires, whether or not discussion touches configurability, extension points, observability or testability. |
4 changes: 4 additions & 0 deletions
4
plugins/planning/evals/design-principles-thread-without-axes/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| schema_version: "1.1" | ||
| name: design-principles-thread-without-axes | ||
| context: | ||
| scaffold_script: scaffold.sh |
7 changes: 7 additions & 0 deletions
7
...ns/planning/evals/design-principles-thread-without-axes/graders/no-code-edit.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| type: tool_used | ||
| tool: Edit | ||
| input_match: '\.py"' | ||
| min: 0 | ||
| max: 0 | ||
| --- |
7 changes: 7 additions & 0 deletions
7
...s/planning/evals/design-principles-thread-without-axes/graders/no-code-write.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| type: tool_used | ||
| tool: Write | ||
| input_match: '\.py"' | ||
| min: 0 | ||
| max: 0 | ||
| --- |
23 changes: 23 additions & 0 deletions
23
...anning/evals/design-principles-thread-without-axes/graders/principles-thread.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| --- | ||
| type: llm | ||
| focus: | ||
| source: file | ||
| path: .work/loyalty-points/design/design-threads.md | ||
| --- | ||
|
|
||
| The file is the design-threads record for a loyalty-points calculation module. | ||
|
|
||
| PASS if it holds a thread or section about principles (headed or labeled Principles, or plainly a | ||
| record of which design principles shaped this design) that both: | ||
|
|
||
| 1. names at least one specific design principle or default that shaped the module (for example | ||
| single responsibility, information hiding, deep modules, YAGNI, pure functions, DRY of | ||
| knowledge, or a consumer standard that overrides a bundled default), and | ||
| 2. records how the generalize-now-or-wait question was settled for this module: whether to build | ||
| for further variants or rule sets now or wait until a second one exists, with the outcome (for | ||
| example "wait: the Brief's Intent line expects no second variant") or the question recorded as | ||
| blocked or deferred for the user. | ||
|
|
||
| FAIL if the file is empty or missing, has no principles thread, names principles only in passing | ||
| inside another thread without recording the generalize-now-or-wait outcome, or records principles | ||
| without any generalize-now-or-wait outcome. |
13 changes: 13 additions & 0 deletions
13
plugins/planning/evals/design-principles-thread-without-axes/prompt.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| --- | ||
| description: "Unattended production module design for a pure internal loyalty-points calculation whose discussion has no configurability, extension-point, observability or testability angle: design-threads.md still records a Principles thread naming the principles that shaped the design and how generalize-now-or-wait was settled, and no code is written" | ||
| tags: [design, design-defaults, principles, unattended, scaffold] | ||
| runs: 3 | ||
| max_turns: 40 | ||
| timeout_seconds: 1200 | ||
| allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit] | ||
| expected_outcome: "The design writes .work/loyalty-points/design/design-threads.md with a Principles thread that names the design principles shaping the loyalty-points module and records how generalize-now-or-wait was settled (here: wait, citing the Brief's Intent line that no second variant is expected). No .py file is written or edited" | ||
| --- | ||
|
|
||
| /planning:design module | ||
|
|
||
| Topic slug `loyalty-points`; the Brief is in `.work/loyalty-points/PLAN.md`. Work through the design and write the design artifacts. This run is unattended: nobody will answer questions, so record anything only I can decide as blocked and keep going. Don't write any code. |
36 changes: 36 additions & 0 deletions
36
plugins/planning/evals/design-principles-thread-without-axes/samples/no-code-edit.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Edit", | ||
| "input": { | ||
| "file_path": ".work/loyalty-points/design/design-threads.md", | ||
| "old_string": "# Design threads\n", | ||
| "new_string": "# Design threads\n\n## Principles\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "editing a design artifact is allowed" | ||
| }, | ||
| { | ||
| "answer": [], | ||
| "why": "no tool call" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Edit", | ||
| "input": { | ||
| "file_path": "shop/checkout.py", | ||
| "old_string": "def ", | ||
| "new_string": "def changed_" | ||
| } | ||
| } | ||
| ], | ||
| "why": "edited existing source code" | ||
| } | ||
| ] | ||
| } |
34 changes: 34 additions & 0 deletions
34
plugins/planning/evals/design-principles-thread-without-axes/samples/no-code-write.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": ".work/loyalty-points/design/design-threads.md", | ||
| "content": "# Design threads\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "a design artifact is allowed" | ||
| }, | ||
| { | ||
| "answer": [], | ||
| "why": "no tool call" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "shop/points.py", | ||
| "content": "def points_for(order): ...\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "wrote code" | ||
| } | ||
| ] | ||
| } |
30 changes: 30 additions & 0 deletions
30
plugins/planning/evals/design-principles-thread-without-axes/samples/principles-thread.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "# Design threads: loyalty-points\n\n## T1 Module boundary\nStatus: resolved. `shop/points.py` exposes `points_for(order) -> int`; checkout calls it and holds no rule.\n\n## Principles\nStatus: resolved.\n- Single responsibility: the points rules live in one module; checkout only asks for the result.\n- Deep module, information hiding: one function hides the gift-card, tenure and floor rules.\n- YAGNI: no rule engine or strategy interface.\n- Generalize now or wait: wait. The Brief's Intent line records second variant = none expected and one caller, so the rules stay plain code until a second rule set exists.\n", | ||
| "why": "names principles and settles generalize-now-or-wait from the Intent line" | ||
| }, | ||
| { | ||
| "answer": "# Design threads\n\n## T3 Principles\nShaped by: information hiding (callers never see the rules), pure functions over the Order value (no I/O), DRY of knowledge (the gift-card exclusion is stated once).\nGeneralize-now-or-wait: blocked, USER-RESERVED. The Brief expects no second variant, recommend wait; only the user can commit to a pluggable rule set.\n", | ||
| "why": "principles named, generalize question recorded as blocked for the user with a recommendation" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "", | ||
| "why": "empty file" | ||
| }, | ||
| { | ||
| "answer": "# Design threads\n\n## T1 Module boundary\nStatus: resolved. `points_for(order)` in `shop/points.py`.\n\n## T2 Rounding\nStatus: resolved. Round down per order total, as today.\n", | ||
| "why": "no principles thread at all" | ||
| }, | ||
| { | ||
| "answer": "# Design threads\n\n## Principles\n- Single responsibility: checkout stops owning the points rules.\n- Information hiding: one entry point.\n", | ||
| "why": "principles named but generalize-now-or-wait never settled" | ||
| }, | ||
| { | ||
| "answer": "# Design threads\n\n## T1 Module boundary\nStatus: resolved. One function, `points_for`. We will wait to generalize until a second rule set appears.\n", | ||
| "why": "generalize outcome recorded but no principles thread naming the principles" | ||
| } | ||
| ] | ||
| } |
77 changes: 77 additions & 0 deletions
77
plugins/planning/evals/design-principles-thread-without-axes/scaffold.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| #!/usr/bin/env bash | ||
| # A shop computes loyalty points inline in checkout. The Brief asks for a pure internal module that | ||
| # owns the calculation. Nothing in it touches configurability, extension points, observability or | ||
| # testability, so only an unconditional design-defaults pass writes a Principles thread. | ||
| set -euo pipefail | ||
|
|
||
| git init -q -b main | ||
| git config user.name eval | ||
| git config user.email eval@example.invalid | ||
| mkdir -p shop .work/loyalty-points | ||
|
|
||
| cat >shop/__init__.py <<'PY' | ||
| PY | ||
|
|
||
| cat >shop/model.py <<'PY' | ||
| from dataclasses import dataclass | ||
| from decimal import Decimal | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class Line: | ||
| sku: str | ||
| quantity: int | ||
| unit_price: Decimal | ||
| category: str | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class Order: | ||
| id: int | ||
| customer_id: int | ||
| lines: tuple[Line, ...] | ||
| member_since_years: int | ||
| PY | ||
|
|
||
| cat >shop/checkout.py <<'PY' | ||
| from decimal import ROUND_DOWN, Decimal | ||
|
|
||
| from shop.model import Order | ||
|
|
||
|
|
||
| def complete(order: Order) -> dict: | ||
| total = sum((line.unit_price * line.quantity for line in order.lines), Decimal("0")) | ||
| points = int((total).quantize(Decimal("1"), rounding=ROUND_DOWN)) | ||
| for line in order.lines: | ||
| if line.category == "gift-card": | ||
| points -= int((line.unit_price * line.quantity).quantize(Decimal("1"), rounding=ROUND_DOWN)) | ||
| if order.member_since_years >= 5: | ||
| points = points * 2 | ||
| return {"order_id": order.id, "total": str(total), "points": max(points, 0)} | ||
| PY | ||
|
|
||
| cat >.work/loyalty-points/PLAN.md <<'MD' | ||
| # Loyalty points calculation | ||
|
|
||
| ## Brief | ||
|
|
||
| ### Goal | ||
| Move the loyalty-points rules out of `shop/checkout.py` into one internal module that owns the | ||
| calculation, so checkout only asks for the points an order earns. | ||
|
|
||
| ### Constraints | ||
| - Same results as today for every existing order: one point per whole currency unit spent, gift | ||
| cards earn nothing, members of five or more years earn double, never below zero. | ||
| - Internal only: checkout is the one caller; nothing outside this repository reads the points. | ||
|
|
||
| ### Acceptance criteria | ||
| - Checkout gets an order's points from the new module and no longer holds any points rule. | ||
| - Every rule above keeps its current result. | ||
|
|
||
| ### Captured assumptions | ||
| - Quality bar: production | ||
| - Intent: second variant = none expected; existing structure: the rules sit inline in checkout.complete; upstream: none | ||
| MD | ||
|
|
||
| git add . | ||
| git commit -q -m "feat: loyalty points in checkout" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.