Skip to content

Build adaptive learning feedback loop from accumulated findings #7

Closed as not planned

Description

@epowell101

Scope note. The hunt feedback portion of this issue — logging which hypotheses produced findings and surfacing productive patterns to the Threat Hunter — is now owned by #430 (agentic hypothesis generation from the hunt ledger), built on the hunt ledger in #427. Treat #430 as the hunt-track slice of this loop and keep this issue to triage feedback and detection feedback. Note also that #174 (systematic analyst feedback loop) was closed not-planned; this issue is the surviving version of that idea.


Issue: Build adaptive learning feedback loop from accumulated findings

Labels: enhancement, orchestration, P2, complexity: L


Problem

Vigil's agents operate statelessly — each triage, investigation, or hunt is independent of previous ones. The Triage agent doesn't learn that certain source IPs consistently produce false positives in this environment. The Threat Hunter doesn't know which hypotheses were productive last week. The Detection Engineering module doesn't track which generated rules actually fired.

Proprietary platforms claim their systems "learn in real-time as your environment changes," refining their behavior based on accumulated probes and responses. This is a meaningful operational advantage: a system that improves with use delivers increasing value over time.

Proposed Solution

Where it lives

  • New service: services/feedback_loop.py
  • Enhancement to: Triage Agent, Threat Hunter Agent, Detection Engineering module
  • New database tables for feedback storage

How it works

  • Triage feedback: When an analyst marks a finding as false positive (or confirms it), log the finding characteristics (source, type, severity, indicators) and the outcome. Over time, build a per-environment false-positive profile that the Triage agent consults before scoring.
  • Hunt feedback: After each threat hunt, log which hypotheses produced findings and which didn't. Surface productive hypothesis patterns to the Threat Hunter for future hunts.
  • Detection feedback: When a newly generated or recommended detection rule fires (or doesn't fire after 30 days), log the outcome. Use this to weight future rule recommendations — rules similar to ones that fired are ranked higher.
  • All feedback is stored locally in PostgreSQL. No data leaves the environment.
  • Feedback is consumed as context injected into agent prompts, not as model fine-tuning.

Interfaces

  • Input: Analyst actions (confirm/dismiss findings), rule activation data (from SIEM via MCP), hunt session outcomes
  • Output: Environment-specific scoring adjustments, hypothesis rankings, rule recommendation weights
  • Connects to: Triage Agent, Threat Hunter Agent, Detection Engineering module, Splunk MCP (for rule activation data)

Acceptance Criteria

  • After 10+ false-positive dismissals of the same finding type, Triage agent reduces severity for that pattern
  • Threat Hunter surfaces previously productive hypotheses first when starting a new hunt
  • Detection rule recommendations are weighted by activation history in the environment
  • All feedback data is stored in PostgreSQL with timestamps and analyst attribution
  • Feedback can be reset per-environment without affecting the base system
  • A /feedback/stats API endpoint shows accumulated feedback metrics
  • Documentation in docs/ADAPTIVE_LEARNING.md
  • Tests in tests/test_feedback_loop.py

Dependencies

  • Requires database migration for feedback tables
  • Requires Splunk MCP integration for rule activation data (existing)
  • Does not depend on other issues in this series

Complexity Estimate

Size: L

Rationale: Touches three agents, requires new database tables, a new service, and a feedback collection mechanism. The prompt injection approach (context, not fine-tuning) keeps it tractable, but the data pipeline — capturing analyst actions, correlating with outcomes, and surfacing actionable context — is a multi-component effort.

Competitive Context

Exposed by: Platforms claiming real-time adaptive learning from environment interaction
Category: orchestration / adaptive-learning
Coverage before: Gap — agents operate statelessly across sessions
Coverage after: Covered — environment-specific feedback improves triage, hunting, and detection over time

Activity

  1. added
    factory:needs-workNot ready for the factory as written; edit the issue to have it re-reviewed
    on Aug 27, 2026
  2. samarmstrong commented on Aug 27, 2026

    @samarmstrong
    Collaborator

    🏭 The factory reviewed this issue and is not picking it up as written.

    Issue #7 should not be built as written. The opening claim — that agents run statelessly across sessions — is already false in this repo, and the proposed services/feedback_loop.py plus new tables would stack a second learning system on top of ones that already exist.

    Triage already has cross-session memory. core/agents/prompts.py tells every agent to mempalace_search entities before work, apply prior decisions instead of re-analyzing, and store false-positive reasoning immediately. The triage record in core/agents/builtins.py names the rooms: agent-decisions/triage-history and false-positives. The daemon already queries that palace on new findings (services/daemon/orchestrator.py _fetch_prior_palace_context) and persists closed investigations into it.

    Analyst outcome capture also already exists, with timestamps and attribution:

    • AIDecisionLog in core/storage/models.py (actual_outcome, human_decision, feedback_comment, human_reviewer, feedback_timestamp)
    • POST /api/ai/decisions/{id}/feedback and GET /api/ai/decisions/stats in core/reporting/ai_decisions_router.py — the Decisions console already uses this (clients/web/src/redesign/screens/decisions/)
    • Case close as false_positive with false_positive_reason (core/cases/case_workflow_service.py, CaseClosureInfo)
    • IOC-level is_false_positive on case_iocs

    The proposed /feedback/stats endpoint duplicates /api/ai/decisions/stats. New feedback tables duplicate ai_decision_logs and case_closure_info. Injecting a “false-positive profile” into prompts duplicates mempalace search, which is the MCP memory the agents already have.

    Detection feedback has the same shape of problem. There is no Vigil-owned rule-recommendation ranker to hang “fired vs 30-day silent” weights on. Detection engineering is the 7,200+ rule catalog plus MCP tools (core/detections/, docs/DETECTION_ENGINEERING.md), including tribal-knowledge tools (log_decision, add_learning, get_relevant_decisions). Hunt feedback was already moved to #430 and that issue is closed; the hunt ledger itself is agent_events (infra/database/init/19_agent_ledger.sql). Building hypothesis ranking here would be a third copy.

    The rest of the issue is the wrong kind of product: a new service, new schema, new API, new doc, and a hardcoded procedure (“after 10+ FP dismissals, reduce severity”). That boxes the agent into a scoring pipeline instead of giving it a tool and a goal. #174 (the earlier “systematic analyst feedback loop”) was closed not-planned; this issue is that idea with more surface area, not a narrower one.

    The real gap, if any, is wiring: analyst FP/TP actions land in AIDecisionLog / case closure and may never be written into mempalace, so triage only learns when it happens to search. That is a capture-to-memory bug, not a new adaptive-learning subsystem. Reframe around that — reuse AIDecisionLog and mempalace, kill any second store — before an agent is asked to implement anything.


    Edit the description and the factory will review it again on a later tick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    factory:needs-workNot ready for the factory as written; edit the issue to have it re-reviewedstate:backlogParked pending the agent overhaultype:feature-requestNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions