feat(hooks): implement prompt-based evaluation

[onatozmenn]

HUMAN:
I reviewed the implementation and test results. The PR is ready for review.

---

AGENT:

Why

HookType.PROMPT is already public, but it currently falls open without evaluating anything. This adds the intended single-completion policy check between command hooks and tool-using agent hooks.

Summary

• Runs prompt hooks with the conversation's current LLM.
• Keeps the policy separate from the untrusted HookEvent payload and parses the shared allow/deny result.
• Tracks timeout, telemetry, cost, and prompt-hook:<name> metrics.
• Fails open with success=False when the model is unavailable or returns an invalid result.
• Adds tests, a runnable example, and matching documentation in docs(hooks): document prompt-based evaluation docs#639.

Issue Number

Closes #2755

How to Test

• Full hook suite: 143 passed on the current branch.
• Changed-file pre-commit checks pass.
• The example was run against a real GitHub Models endpoint: the test command was allowed and the destructive command was denied. The output is in .pr/prompt-hooks-live-run.md.
• The docs example check passes against the open docs branch. It will remain red in GitHub until docs(hooks): document prompt-based evaluation docs#639 is merged or the workflow can check that fork branch.

Type

• Bug fix
• Feature
• Refactor
• Breaking change
• Docs / chore

Notes

GitHub Copilot assisted with this PR.

This first version uses the active conversation LLM and the HookEvent payload only. Per-hook profile overrides and extra conversation or workspace context are left for follow-up; HookType.AGENT remains the option when tools or workspace investigation are needed.

[onatozmenn]

Maintainers: this changes LLM-backed hook decision behavior. Please add the integration-test label before the PR is marked ready, per the repository testing guidance.

[enyst]

Thank you for the PR!

Could you perhaps add some logs from the real-world example run, to a temporary (but committed) .pr/ directory? Generally, we have a convention that .pr/ directory is good sometimes, for artefacts that show or prove the PR's behavior, maybe with a .md file too to document results.

[enyst]

@OpenHands Do an initial /codereview of this PR. Post your feedback directly with gh api and event as you think appropriate.

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[smolpaws]

Initial review complete. I did not find a code-level correctness blocker in the prompt-hook implementation: policy/event separation, explicit fail-open signaling, current-LLM lookup, API routing, isolated metrics, lifecycle behavior, and REST compatibility are all covered coherently.

Local verification on the exact PR head (b4b21c12):

• 185 passed, 66 skipped: hooks + hook REST router + REST breakage checker + example discovery
• 6 passed: OpenAPI discriminator tests
• changed-file pre-commit: Ruff, pycodestyle, Pyright, dependency rules all passed

I’m leaving COMMENT rather than APPROVE because this changes LLM-backed agent policy behavior and still lacks the required evaluation evidence:

• integration-test label is absent. I attempted to add it, but the current authenticated account lacks label permissions.
• The requested live-provider .pr/ run artifact is absent, and this environment has no LLM_API_KEY, so I could not produce honest real-world logs.
• check-examples fails only because 55_prompt_hooks/main.py is not yet present on docs main; OpenHands/docs#639 contains the correct synchronized example block but is still open.

Residual follow-up, not introduced by this PR: hook callbacks are synchronous, so command/agent/prompt hooks can block an arun() event loop while evaluating. Fixing that requires an async hook pipeline rather than a prompt-hook-only patch.

Profile selection is addressed in the inline thread.

This review was created by an AI agent (OpenHands) on behalf of Engel Nyst.

[VascoSch92]

Just as a note: we will need also a PR in the docs to document the Hook

[onatozmenn]

Added the live run output in .pr/prompt-hooks-live-run.md. I ran the example against GitHub Models: the test command was allowed and the destructive command was denied.

The docs PR is already open at OpenHands/docs#639.