Repository navigation
Description
Scope note. The hunt feedback portion of this issue — logging which hypotheses produced findings and surfacing productive patterns to the Threat Hunter — is now owned by #430 (agentic hypothesis generation from the hunt ledger), built on the hunt ledger in #427. Treat #430 as the hunt-track slice of this loop and keep this issue to triage feedback and detection feedback. Note also that #174 (systematic analyst feedback loop) was closed not-planned; this issue is the surviving version of that idea.
Issue: Build adaptive learning feedback loop from accumulated findings
Labels: enhancement, orchestration, P2, complexity: L
Problem
Vigil's agents operate statelessly — each triage, investigation, or hunt is independent of previous ones. The Triage agent doesn't learn that certain source IPs consistently produce false positives in this environment. The Threat Hunter doesn't know which hypotheses were productive last week. The Detection Engineering module doesn't track which generated rules actually fired.
Proprietary platforms claim their systems "learn in real-time as your environment changes," refining their behavior based on accumulated probes and responses. This is a meaningful operational advantage: a system that improves with use delivers increasing value over time.
Proposed Solution
Where it lives
- New service:
services/feedback_loop.py - Enhancement to: Triage Agent, Threat Hunter Agent, Detection Engineering module
- New database tables for feedback storage
How it works
- Triage feedback: When an analyst marks a finding as false positive (or confirms it), log the finding characteristics (source, type, severity, indicators) and the outcome. Over time, build a per-environment false-positive profile that the Triage agent consults before scoring.
- Hunt feedback: After each threat hunt, log which hypotheses produced findings and which didn't. Surface productive hypothesis patterns to the Threat Hunter for future hunts.
- Detection feedback: When a newly generated or recommended detection rule fires (or doesn't fire after 30 days), log the outcome. Use this to weight future rule recommendations — rules similar to ones that fired are ranked higher.
- All feedback is stored locally in PostgreSQL. No data leaves the environment.
- Feedback is consumed as context injected into agent prompts, not as model fine-tuning.
Interfaces
- Input: Analyst actions (confirm/dismiss findings), rule activation data (from SIEM via MCP), hunt session outcomes
- Output: Environment-specific scoring adjustments, hypothesis rankings, rule recommendation weights
- Connects to: Triage Agent, Threat Hunter Agent, Detection Engineering module, Splunk MCP (for rule activation data)
Acceptance Criteria
- After 10+ false-positive dismissals of the same finding type, Triage agent reduces severity for that pattern
- Threat Hunter surfaces previously productive hypotheses first when starting a new hunt
- Detection rule recommendations are weighted by activation history in the environment
- All feedback data is stored in PostgreSQL with timestamps and analyst attribution
- Feedback can be reset per-environment without affecting the base system
- A
/feedback/statsAPI endpoint shows accumulated feedback metrics - Documentation in
docs/ADAPTIVE_LEARNING.md - Tests in
tests/test_feedback_loop.py
Dependencies
- Requires database migration for feedback tables
- Requires Splunk MCP integration for rule activation data (existing)
- Does not depend on other issues in this series
Complexity Estimate
Size: L
Rationale: Touches three agents, requires new database tables, a new service, and a feedback collection mechanism. The prompt injection approach (context, not fine-tuning) keeps it tractable, but the data pipeline — capturing analyst actions, correlating with outcomes, and surfacing actionable context — is a multi-component effort.
Competitive Context
Exposed by: Platforms claiming real-time adaptive learning from environment interaction
Category: orchestration / adaptive-learning
Coverage before: Gap — agents operate statelessly across sessions
Coverage after: Covered — environment-specific feedback improves triage, hunting, and detection over time
Activity
samarmstrong commentedon Aug 27, 2026
🏭 The factory reviewed this issue and is not picking it up as written.
Issue #7 should not be built as written. The opening claim — that agents run statelessly across sessions — is already false in this repo, and the proposed
services/feedback_loop.pyplus new tables would stack a second learning system on top of ones that already exist.Triage already has cross-session memory.
core/agents/prompts.pytells every agent tomempalace_searchentities before work, apply prior decisions instead of re-analyzing, and store false-positive reasoning immediately. The triage record incore/agents/builtins.pynames the rooms:agent-decisions/triage-historyandfalse-positives. The daemon already queries that palace on new findings (services/daemon/orchestrator.py_fetch_prior_palace_context) and persists closed investigations into it.Analyst outcome capture also already exists, with timestamps and attribution:
AIDecisionLogincore/storage/models.py(actual_outcome,human_decision,feedback_comment,human_reviewer,feedback_timestamp)POST /api/ai/decisions/{id}/feedbackandGET /api/ai/decisions/statsincore/reporting/ai_decisions_router.py— the Decisions console already uses this (clients/web/src/redesign/screens/decisions/)false_positivewithfalse_positive_reason(core/cases/case_workflow_service.py,CaseClosureInfo)is_false_positiveoncase_iocsThe proposed
/feedback/statsendpoint duplicates/api/ai/decisions/stats. New feedback tables duplicateai_decision_logsandcase_closure_info. Injecting a “false-positive profile” into prompts duplicates mempalace search, which is the MCP memory the agents already have.Detection feedback has the same shape of problem. There is no Vigil-owned rule-recommendation ranker to hang “fired vs 30-day silent” weights on. Detection engineering is the 7,200+ rule catalog plus MCP tools (
core/detections/,docs/DETECTION_ENGINEERING.md), including tribal-knowledge tools (log_decision,add_learning,get_relevant_decisions). Hunt feedback was already moved to #430 and that issue is closed; the hunt ledger itself isagent_events(infra/database/init/19_agent_ledger.sql). Building hypothesis ranking here would be a third copy.The rest of the issue is the wrong kind of product: a new service, new schema, new API, new doc, and a hardcoded procedure (“after 10+ FP dismissals, reduce severity”). That boxes the agent into a scoring pipeline instead of giving it a tool and a goal. #174 (the earlier “systematic analyst feedback loop”) was closed not-planned; this issue is that idea with more surface area, not a narrower one.
The real gap, if any, is wiring: analyst FP/TP actions land in
AIDecisionLog/ case closure and may never be written into mempalace, so triage only learns when it happens to search. That is a capture-to-memory bug, not a new adaptive-learning subsystem. Reframe around that — reuseAIDecisionLogand mempalace, kill any second store — before an agent is asked to implement anything.Edit the description and the factory will review it again on a later tick.
[SW C6/S4] Wire it: sensor -> redactor -> engine -> router R1 -> store (
[SW C6/S7b] Helm poll wiring, NetworkPolicy probe fix, Down threshold…