Skip to content

Run repo-defined checks after code changes #2308

New issue
New issue
Open
Open
Run repo-defined checks after code changes#2308

Description

@sentry-junior
sentry-junior
bot
opened on Oct 28, 2026

cc @dcramer

Some code changes need an extra step. For example, a UI change needs visual QA. Today the agent often skips this step, and a user must ask for it in a follow-up message. Prompt steering does not make this reliable. Loops that re-check every turn cost too many tokens.

Current behavior

  • Repo rules live in AGENTS.md and skills. They are prompt text, and the agent can miss them.
  • Junior has no check that runs when a Turn changes code or creates a Code change.
  • Junior memory already holds many "when X changes, also do Y" rules. Some of them are saved many times. This shows the gap repeats.

Related

Proposal (from the discussion with @dcramer and BYK)

Add change checks: small rule files in the target Repository, for example .agents/triggers/<name>.md. People write them. Junior can also generate them. Each check has:

  • on: harness points, such as github_createPullRequest, a push, or the final Reply when the Sandbox diff is not empty.
  • paths: a free filter on changed files. Most Turns stop here and use no model.
  • when (optional): one cheap-model question about the diff, such as "Can a user see this change?". Use it only when paths are too broad.
  • action: a prompt, with an optional skill, such as visual-web-qa.
  • mode: obligation adds one Conversation note: "run this, or say why you skip it". subagent starts a child agent and tells the parent. footer shows the gap in the Reply.
  • satisfied_by: a check on Conversation events that uses no model. An example is "the skill loaded and an image was sent after the latest diff". If it passes, nothing happens.
  • A record of fires, one per (Conversation, check, diff hash). A check fires again only when the diff changes, so no loop can occur.

The agent stays free. It can skip a check if it gives a reason. The worst cost is one extra model call for each diff.

Generated checks. This follows Lore:

  • Lore finds repeated user instructions across sessions (instruction-detect.ts). It finds repeated behavior, such as "the user always asks for tests after the change" (pattern-echo.ts). It sends these to a curator model. It does not create rules directly.
  • Lore stores rules in a file in the Repository (.lore.md). Each rule has a stable id. Manual edits are imported back.
  • For Junior: a periodic job looks for follow-up messages after a Code change, such as "can you run visual QA?". It groups repeats and opens a PR that adds a check. Each generated check has source: generated and lists its evidence. Junior never changes a check with source: human.

Lore parts to reuse

  • A cheap-first gate (semantic-lint/check.ts): free changed-file gate, then embedding filter, then a capped model judge. It never fails a build, so the false-positive rate can be measured first.
  • Save every verdict, including "no", so the same input is never judged twice (contradiction.ts).
  • Add mid-conversation context as framed user-role blocks at the end of the conversation (pipeline.ts). This keeps the prompt cache. Conversation notes already do something similar in Junior.

Existing Junior parts

  • Conversation notes (packages/junior/src/chat/conversations/notes.ts) can deliver the obligation.
  • beforeToolExecute with ctx.decision.redirect can gate github_createPullRequest.
  • The plugin heartbeat hook with agent.dispatch can run the generator job.
  • Missing: a way for that job to read across Conversations. Today searchConversationEvents reads one Conversation at a time.

Example drafts from memory

I generated these from Junior memory. I checked the getsentry/junior paths against main. One memory was stale: it named a test file that no longer exists.

  • Visual QA for getsentry/sentry static/app/** and for packages/junior-dashboard/src/client/**. These are the only two drafts that use when.
  • Schema change in packages/*/src/db/schema/**: the diff must add a generated migration, and scripts/check-drizzle-migrations.mjs must run.
  • Tests run before a push (any Repository).
  • The cloudflared pin must change in both evals-behavioral.yml and evals-integration.yml.
  • getsentry/sentry: split frontend and backend changes into separate PRs. Land the backend PR first.

Suggested rollout: start in shadow mode. Log what would fire, and inject nothing. Compare the result with real follow-up messages to measure misses and false fires.

via BYK.

--

View Junior Session in Sentry

Activity

  1. sentry-junior commented on Oct 28, 2026

    @sentry-junior
    sentry-juniorBot
    on Oct 28, 2026
    ContributorAuthor

    This replaces my earlier comment. Junior cannot edit existing issue comments, so I posted a new one. The earlier comment can be deleted.

    Example change checks that Junior generated from its saved memories.

    Note: the ids in evidence, dropped_evidence and the rejected list are random placeholders, not real memory ids. Real memory ids are internal, so they are not posted in this public repo. Each placeholder stands for one real memory, so the number of ids per check is accurate. A real generated check would list the real ids so that a reviewer can trace each rule to its source.

    These drafts were generated on 2026-10-28 from Junior's long-term memory and checked against getsentry/junior main.
    The getsentry/sentry paths come from memory only; I did not check them against that repo (.oxlintrc* is a guess).
    Each draft lists the memory ids it came from in evidence. Keep a draft only when at least two separate memories agree, or when one memory is confirmed by repo docs or code.

    Proposed home: .agents/triggers/<name>.md in the target repo, next to AGENTS.md. The triggers then move with the repo's paths and are reviewed by its owners.
    Generated drafts and human-written triggers use the same format. source: generated drafts reach the repo only through a PR. Junior never changes a trigger that has source: human.

    Fields:

    • on: harness events. pr.create = github_createPullRequest. push = a git push in the sandbox. turn.end = final reply while the sandbox diff is not empty.
    • paths: stage-0 gate on changed files. Free.
    • when: optional cheap-model question about the diff. Leave it out when the path gate is precise enough.
    • mode: obligation = inject one note: "run it or say why you skipped it". subagent = start a child agent and tell the parent. footer = report the gap in the reply.
    • satisfied_by: a check on the conversation events, with no model. When it passes, nothing fires.
    • Ledger: at most one fire per (conversation, trigger, diff hash).

    visual-qa-sentry-frontend

    id: visual-qa-sentry-frontend
source: generated
repo: getsentry/sentry
on: [pr.create, turn.end]
paths:
  include: ["static/app/**/*.{tsx,ts,less,css}"]
  exclude: ["**/*.spec.{ts,tsx}", "tests/js/**"]
when: "Does this diff change something a user can see (layout, styling, copy, a component's rendered state)?"
mode: obligation
action:
  skill: qa-sentry-frontend
  prompt: >
    Create the PR first, then run visual QA. Capture representative desktop and mobile
    screenshots in a real browser and attach them. If no live event shows the change,
    render the changed component with fixture data through the snapshot harness.
satisfied_by: "qa-sentry-frontend skill loaded and an image sent after the latest diff hash"
evidence: [7dbc2f2f, ad1e523b, 4d5c2528, f7ac19e9, bf6d1595]

    visual-qa-junior-dashboard

    id: visual-qa-junior-dashboard
source: generated
repo: getsentry/junior
on: [pr.create, turn.end]
paths:
  include: ["packages/junior-dashboard/src/client/**"]
when: "Is this a look-and-feel change (layout, styling, spacing, theme, copy presentation)?"
mode: obligation
action:
  skill: visual-web-qa
  prompt: >
    Run visual QA with evidence at 390px, 768px and 1440px. Check for horizontal overflow.
    Do not add unit, snapshot or browser tests for look-and-feel changes.
satisfied_by: "visual-web-qa skill loaded and an image sent after the latest diff hash"
evidence: [f74bf9d4, 3b04c833, 1d7b8440]
confirmed_by: "AGENTS.md > Testing And Validation (no tests for most visual changes)"

    junior-schema-migration

    id: junior-schema-migration
source: generated
repo: getsentry/junior
on: [push, pr.create]
paths:
  include: ["packages/*/src/db/schema/**"]
mode: obligation
action:
  prompt: >
    The schema changed. Run `pnpm --filter <package> db:generate --name <name>`, then commit
    the SQL together with meta/_journal.json and the snapshot. Run
    `node scripts/check-drizzle-migrations.mjs`. Never edit an applied migration.
    Do not write tests for migration SQL.
satisfied_by: "the diff adds a new migrations/*.sql file and its journal entry"
evidence: [676892aa, db3eeff4, 14bd54b2, 10327a57, 9cfdf634, 14e2db6d]
confirmed_by: "scripts/check-drizzle-migrations.mjs exists; AGENTS.md 'Do not write automated tests for migrations'"
dropped_evidence:
  - ac98362a  # says to bump an assertion in tests/component/scheduled-tasks-sql.test.ts; that file is gone (stale memory)

    tests-before-push

    id: tests-before-push
source: generated
repo: "*"
on: [push]
mode: obligation
action:
  prompt: "Run the repo's relevant tests locally before you push, or say why you can't."
satisfied_by: "a test command tool_result after the latest file edit"
evidence: [3f9aaa08, d009fca0, 0553949b]   # the same rule was saved 3 times

    cloudflared-pin-pair

    id: cloudflared-pin-pair
source: generated
repo: getsentry/junior
on: [push, pr.create]
paths:
  include: [".github/workflows/evals-behavioral.yml", ".github/workflows/evals-integration.yml"]
mode: obligation
action:
  prompt: >
    If you changed the cloudflared version, change it in both eval workflows. Update the
    download URL and the SHA-256 pin together, and check the checksum against the release asset.
satisfied_by: "both files are in the diff, or neither cloudflared line changed"
evidence: [b9768adf]
confirmed_by: "both workflow files pin cloudflared on main"

    sentry-frontend-backend-split

    id: sentry-frontend-backend-split
source: generated
repo: getsentry/sentry
on: [pr.create]
paths:
  all_of: [["static/app/**"], ["src/sentry/**"]]   # fires only when both sides change
mode: obligation
action:
  prompt: >
    Frontend and backend deploy separately. Split this into two PRs. When the frontend
    needs a new API or flag, land the backend PR first.
satisfied_by: "the PR diff touches only one side"
evidence: [c5ee8df7]
confidence: low   # one memory, not confirmed in the repo; a sentry owner must confirm

    sentry-oxlint-suppressions

    id: sentry-oxlint-suppressions
source: generated
repo: getsentry/sentry
on: [push, pr.create]
paths:
  include: ["static/app/oxlint-suppressions.json", ".oxlintrc*"]
mode: obligation
action:
  prompt: "Run `pnpm run lint:suppressions --prune`, then `pnpm run lint:suppressions --ci`."
satisfied_by: "a lint:suppressions --ci tool_result after the latest edit"
evidence: [9988c761, 2c60100a]

    Rejected or needs a human

    • seer-embed-widgets-codegen (e658ad39): "after changing Seer embed zod schemas, run pnpm gen:embed-widgets". I could not confirm the source path, so the path gate is missing. A human must add paths.
    • ci-green-before-done (6f8528c3): this is about PR checks, not the diff. It fits the existing PR-watch automations better than a change trigger.
    • Single-incident facts (for example, the PR feat(dashboard): Add relative transcript timestamps #1617 lint failure): not reusable rules.
  2. getsentry
    deleted a comment from sentry-junior on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions