Security: Agent self-modification patterns in Canvas memory module (ASI10) #7918

Activity

1. dockfixlabs commented on Jul 6, 2026

   

   Dockfix Labs (dockfixlabs)

   on Jul 6, 2026

   Author

   More actions

   Excellent classification model, Massimiliano Brighindi brighissimo@gmail.com (@Tuttotorna). The three-tier typology is precisely the right lens:

   • append_only_observation — Read-only state, safe by construction
   • mutable_working_state — Edit-in-place, trust-scoped
   • governance_state_change — Model modification, requires cryptographic verification

   This mapping is clean, implementable, and directly actionable. A question for deeper discussion:

   Does AutoGen's current memory architecture distinguish between these classes? From what I can see scanning the codebase, the Memory abstraction and ChatCompletionContext don't appear to enforce different write semantics per memory type. Everything looks like a flat key-value store with uniform mutation access — which means any agent with update_context permission can effectively perform a governance_state_change without the verification step.

   Two relevant detection approaches from AgentGuard (our ASI scanner):

   1. ASI-AGENT-COLLUSION — Detects patterns where agents manipulate shared state to influence sibling agent behavior across trust boundaries. A governance_state_change without verifiable attestation is exactly the signal this rule looks for at the state-mutation layer.

   2. ASI-MEMORY-POISON — Flags writes to agent memory surfaces that originate from untrusted tool output or inter-agent channels. If Canvas memory can be modified by tool results without integrity checks, this triggers.

   Suggested deterministic guard approach: Wrap mutable memory surfaces in a MemoryGuard that enforces:

   • �ppend_only surfaces: hash-chained, no deletion, write-once
   • mutable surfaces: per-agent namespace isolation + atomic CAS writes
   • governance surfaces: require a signed attestation from an authorized transition function before mutation

   The key insight is that you don't need probabilistic AI to detect this — you need deterministic state-machine invariants. Agent self-modification is detectable at the permission layer by enforcing differential write authority per memory class.

   Would be interested in the team's thoughts on whether a memory-class-aware access control model is on the roadmap.
2. dockfixlabs commented on Jul 16, 2026

   

   Dockfix Labs (dockfixlabs)

   on Jul 16, 2026

   Author

   More actions

   Following up on this one as well — the three-tier enforcement boundary model articulated by Massimiliano Brighindi brighissimo@gmail.com (@Tuttotorna) is a strong foundation, and it would be worth keeping the discussion moving.

   The agreed enforcement boundary:

   • append_only_observation → write-once, hash-chained, no deletion. Safe audit/event memory.
   • mutable_working_state → scoped namespace + CAS/version check. Edit-in-place within trust boundaries.
   • governance_state_change → denied by default unless an authorized transition record exists externally.

   The critical insight from the last exchange is the shared-write-path problem: if tool results and sibling-agent messages can update working memory through the same path as governance state mutations, then an agent can perform a governance_state_change through what appears to be ordinary memory mutation. That's the thing to lock down.

   The fix doesn't require probabilistic AI detection — it's a deterministic state-machine invariant at the permission layer. Differential write authority per memory class is the right primitive.

   Has the team considered adding memory-class-aware access control to their security roadmap? Even without a full implementation, documenting the three-tier boundary in the security model would help users understand which memory surfaces are safe for untrusted agent output and which require attestation.

   Happy to contribute if there's interest.
3. bloxfruitvaluez-lgtm commented on Jul 24, 2026

   

   bloxy (bloxfruitvaluez-lgtm)

   on Jul 24, 2026

   Last edited by bloxfruitvaluez-lgtm

   More actions

   I'm experiencing the same issue as well.

   Environment:

   AutoGen version: [your version]
   Python: [your version]
   OS: [Windows/macOS/Linux]

   I can consistently reproduce the behavior using the same steps described in this issue. I also tested with a clean environment and the problem still occurs.

   This doesn't appear to be an isolated setup issue. Please let me know if you need any additional logs or if there's a workaround available.

   Same Issue is Ocuring with my friend on his Blox Fruit Trading Website.
4. IgorGanapolsky commented on Jul 24, 2026

   

   Igor Ganapolsky (IgorGanapolsky)

   on Jul 24, 2026

   More actions

   Agentic systems are turning into always-on factories: one request fans out into model calls, tool calls, policy checks, and storage.

   The hard part is rarely tokens/sec — it is control-plane reliability:

   1. Fan-out control — hard cap on tools/steps per request before external writes
   2. Durable id + idempotency — one write per business object (ledger / preflight)
   3. Observable long jobs — progress + cooperative cancel on multi-minute runs
   4. Expected vs actual — durable artifact id before marking success
   5. Policy before action — gate that can refuse known-bad side effects

   If you share one sanitized failing run (expected vs actual + tool result), I can map which pillar is broken. Architecture only.