Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   appIssues related to the Codex desktop appIssues related to the Codex desktop app

   subagentIssues involving subagents or multi-agent featuresIssues involving subagents or multi-agent features

   tool-callsIssues related to tool callingIssues related to tool calling

   mcpIssues related to the use of model context protocol (MCP) serversIssues related to the use of model context protocol (MCP) servers

   on May 25, 2026
2. github-actions commented on May 25, 2026

   

   github-actionsBot

   on May 25, 2026 – with GitHub Actions

   Contributor

   More actions

   Potential duplicates detected. Please review them and close your issue if it is a duplicate.

   • Stale Codex subagents #23700
   • Codex app: subagent close request triggers "agent loop died unexpectedly" and repeated message submit failures #23971

   Powered by Codex Action
3. cwolff-oai commented on May 25, 2026

   

   cwolff-oai

   on May 25, 2026

   More actions

   Thanks. Ill review #23700 and #23971 and close this if its a duplicate.
4. jshaofa-ui commented on May 25, 2026

   

   jshaofa-ui

   on May 25, 2026

   More actions

   Fix: multi_agent_v1.close_agent Hangs for Hours When Closing an Unresponsive Subagent

   Issue: openai/codex #24389
   Severity: High — blocks parent agent indefinitely
   Component: codex-rs/core/src/agent/control.rs — AgentControl::shutdown_live_agent
   Labels: multi-agent, hang, timeout, shutdown

   ---

   1. Root Cause Analysis

   1.1 The Problem

   When close_agent is called on an unresponsive subagent, the call can hang for hours (or indefinitely). The root cause is an unbounded await in the shutdown path:

   // codex-rs/core/src/agent/control.rs — shutdown_live_agent()
   pub(crate) async fn shutdown_live_agent(&self, agent_id: ThreadId) -> CodexResult<String> {
       let state = self.upgrade()?;
       let result = if let Ok(thread) = state.get_thread(agent_id).await {
           thread.codex.session.ensure_rollout_materialized().await;
           thread.codex.session.flush_rollout().await?;
           let result = if matches!(thread.agent_status().await, AgentStatus::Shutdown) {
               Ok(String::new())
           } else {
               state.send_op(agent_id, Op::Shutdown {}).await
           };
           thread.wait_until_terminated().await;  // <-- NO TIMEOUT: hangs indefinitely
           result
       } else {
           state.send_op(agent_id, Op::Shutdown {}).await
       };
       let _ = state.remove_thread(&agent_id).await;
       self.state.release_spawned_thread(agent_id);
       result
   }

   1.2 Why It Hangs

   The call chain is:

   1. close_agent(agent_id) → shutdown_agent_tree(agent_id) → shutdown_live_agent(agent_id)
   2. shutdown_live_agent sends Op::Shutdown to the subagent's submission channel
   3. Then calls thread.wait_until_terminated().await — which awaits session_loop_termination
   4. session_loop_termination is a Shared<BoxFuture<'static, ()>> wrapping a JoinHandle<()>
   5. The JoinHandle only completes when the submission_loop task exits

   The submission_loop (in codex-rs/core/src/session/handlers.rs) processes operations sequentially:

   pub(super) async fn submission_loop(
       sess: Arc<Session>,
       config: Arc<Config>,
       rx_sub: Receiver<Submission>,
   ) {
       while let Ok(sub) = rx_sub.recv().await {
           // ... processes each op sequentially ...
           Op::Shutdown => shutdown(&sess, sub.id.clone()).await,  // only reached after current op completes
       }
   }

   If the session loop is currently:

   • Waiting for a model API response that never returns (network hang)
   • Executing a long-running tool call
   • Blocked on a permission prompt awaiting user input
   • Deadlocked on a mutex or channel

   Then Op::Shutdown sits in the channel queue behind the current operation, and wait_until_terminated() blocks forever.

   1.3 Impact Scope

   Component	Affected	Description
   close_agent tool (v1)	Yes	Calls shutdown_live_agent
   close_agent tool (v2)	Yes	Same code path
   shutdown_agent_tree	Yes	Cascades to all descendants
   shutdown_live_agent	Yes	Direct culprit
   CodexThread::shutdown_and_wait	Yes	Same unbounded await pattern
   Session::shutdown_and_wait	Yes	Same unbounded await pattern

   1.4 Why Tests Do Not Catch This

   The test infrastructure uses ThreadManager::with_models_provider_for_tests() which operates in a test mode where:

   • The session loop is mocked/simulated
   • captured_ops() records ops without actually running a real session loop
   • Shutdown completes immediately because there is no real blocking I/O

   // multi_agents_tests.rs — uses mock infrastructure
   fn thread_manager() -> ThreadManager {
       ThreadManager::with_models_provider_for_tests(
           CodexAuth::from_api_key("dummy"),
           built_in_model_providers(...)["openai"].clone(),
       )
   }

   ---

   2. Step-by-Step Fix Approach

   2.1 Design Goals

   1. Add a configurable timeout to shutdown_live_agent so it never blocks indefinitely
   2. Force-terminate the session loop task if the graceful shutdown times out
   3. Log warnings when timeouts occur so operators can investigate the root cause
   4. Maintain backward compatibility — graceful shutdown should still be the default path
   5. Add a config option so users can tune the timeout

   2.2 Implementation Steps

   Step 1: Add shutdown_timeout_ms to MultiAgentV2Config

   File: codex-rs/core/src/config/mod.rs

   // In MultiAgentV2Config struct (around line 995):
   pub struct MultiAgentV2Config {
       pub max_concurrent_threads_per_session: usize,
       pub min_wait_timeout_ms: i64,
       pub max_wait_timeout_ms: i64,
       pub default_wait_timeout_ms: i64,
       pub shutdown_timeout_ms: i64,  // NEW: timeout for graceful agent shutdown
       pub usage_hint_enabled: bool,
       pub usage_hint_text: Option<String>,
       pub root_agent_usage_hint_text: Option<String>,
       pub subagent_usage_hint_text: Option<String>,
       pub tool_namespace: Option<String>,
       pub hide_spawn_agent_metadata: bool,
       pub non_code_mode_only: bool,
   }
   
   // Default (around line 1010):
   impl Default for MultiAgentV2Config {
       fn default() -> Self {
           Self {
               max_concurrent_threads_per_session: DEFAULT_MULTI_AGENT_V2_MAX_CONCURRENT_THREADS_PER_SESSION,
               min_wait_timeout_ms: DEFAULT_MULTI_AGENT_V2_MIN_WAIT_TIMEOUT_MS,
               max_wait_timeout_ms: DEFAULT_MULTI_AGENT_V2_MAX_WAIT_TIMEOUT_MS,
               default_wait_timeout_ms: DEFAULT_MULTI_AGENT_V2_DEFAULT_WAIT_TIMEOUT_MS,
               shutdown_timeout_ms: DEFAULT_SHUTDOWN_TIMEOUT_MS,  // NEW
               usage_hint_enabled: true,
               usage_hint_text: None,
               root_agent_usage_hint_text: None,
               subagent_usage_hint_text: None,
               tool_namespace: None,
               hide_spawn_agent_metadata: false,
               non_code_mode_only: false,
           }
       }
   }
   
   // Add constant near other defaults (around line 100):
   const DEFAULT_SHUTDOWN_TIMEOUT_MS: i64 = 30_000; // 30 seconds

   Step 2: Store the JoinHandle in Codex for forceful termination

   File: codex-rs/core/src/session/mod.rs

   // Change SessionLoopTermination to also store the JoinHandle:
   pub(crate) type SessionLoopTermination = Shared<BoxFuture<'static, ()>>;
   
   pub struct Codex {
       pub(crate) tx_sub: Sender<Submission>,
       pub(crate) rx_event: Receiver<Event>,
       pub(crate) agent_status: watch::Receiver<AgentStatus>,
       pub(crate) session: Arc<Session>,
       pub(crate) session_loop_termination: SessionLoopTermination,
       // NEW: Store the JoinHandle for forceful abort capability
       pub(crate) session_loop_handle: Option<tokio::task::JoinHandle<()>>,
   }

   Update Codex::spawn to store the handle:

   // Around line 662 in session/mod.rs:
   let session_loop_handle = tokio::spawn(async move {
       submission_loop(sess.clone(), config, rx_sub).await;
   });
   
   // Store the handle alongside the termination future:
   session_loop_termination: session_loop_termination_from_handle(session_loop_handle.clone()),
   session_loop_handle: Some(session_loop_handle),

   Step 3: Add wait_until_terminated_with_timeout to CodexThread

   File: codex-rs/core/src/codex_thread.rs

   use std::time::Duration;
   use tokio::time::timeout;
   
   impl CodexThread {
       /// Wait until the session loop terminates, with a timeout.
       /// Returns Ok(()) if terminated normally, Err(()) if timed out.
       pub async fn wait_until_terminated_with_timeout(&self, timeout_duration: Duration) -> Result<(), ()> {
           let termination = self.codex.session_loop_termination.clone();
           match timeout(timeout_duration, termination).await {
               Ok(()) => Ok(()),
               Err(_) => {
                   tracing::warn!(
                       thread_id = %self.codex.session.conversation_id,
                       timeout_ms = timeout_duration.as_millis(),
                       "session loop did not terminate within timeout"
                   );
                   Err(())
               }
           }
       }
   
       /// Forcefully abort the session loop task. This is a last-resort measure
       /// when graceful shutdown has timed out.
       pub fn abort_session_loop(&self) {
           if let Some(ref handle) = self.codex.session_loop_handle {
               if !handle.is_finished() {
                   tracing::warn!(
                       thread_id = %self.codex.session.conversation_id,
                       "forcefully aborting unresponsive session loop"
                   );
                   handle.abort();
               }
           }
       }
   }

   Step 4: Update shutdown_live_agent with timeout and fallback

   File: codex-rs/core/src/agent/control.rs

   use std::time::Duration;
   use tokio::time::timeout;
   
   impl AgentControl {
       /// Submit a shutdown request for a live agent without marking it explicitly closed in
       /// persisted spawn-edge state.
       ///
       /// If the agent does not shut down within the configured timeout, the session loop
       /// is forcefully aborted and cleanup proceeds anyway.
       pub(crate) async fn shutdown_live_agent(&self, agent_id: ThreadId) -> CodexResult<String> {
           let state = self.upgrade()?;
           let result = if let Ok(thread) = state.get_thread(agent_id).await {
               thread.codex.session.ensure_rollout_materialized().await;
               thread.codex.session.flush_rollout().await?;
   
               let result = if matches!(thread.agent_status().await, AgentStatus::Shutdown) {
                   Ok(String::new())
               } else {
                   state.send_op(agent_id, Op::Shutdown {}).await
               };
   
               // NEW: Wait with timeout, then force abort if needed
               let shutdown_timeout = self.get_shutdown_timeout(&thread).await;
               match thread
                   .wait_until_terminated_with_timeout(shutdown_timeout)
                   .await
               {
                   Ok(()) => {
                       tracing::debug!(
                           thread_id = %agent_id,
                           "agent shut down gracefully"
                       );
                   }
                   Err(()) => {
                       // Graceful shutdown timed out — force abort
                       tracing::warn!(
                           thread_id = %agent_id,
                           timeout_ms = shutdown_timeout.as_millis(),
                           "agent shutdown timed out, force-aborting session loop"
                       );
                       thread.abort_session_loop();
                       // Wait briefly for the abort to take effect
                       let _ = thread
                           .wait_until_terminated_with_timeout(Duration::from_secs(5))
                           .await;
                   }
               }
   
               result
           } else {
               state.send_op(agent_id, Op::Shutdown {}).await
           };
   
           let _ = state.remove_thread(&agent_id).await;
           self.state.release_spawned_thread(agent_id);
           result
       }
   
       /// Get the configured shutdown timeout for a thread.
       async fn get_shutdown_timeout(&self, thread: &crate::CodexThread) -> Duration {
           let config = thread.codex.session.get_config().await;
           let timeout_ms = config.multi_agent_v2.shutdown_timeout_ms;
           Duration::from_millis(timeout_ms.max(1000) as u64) // minimum 1 second
       }
   }

   Step 5: Add config parsing for shutdown_timeout_ms

   File: codex-rs/core/src/config/mod.rs

   Add to MultiAgentV2ConfigToml and the parsing function:

   // In MultiAgentV2ConfigToml struct:
   pub struct MultiAgentV2ConfigToml {
       // ... existing fields ...
       pub shutdown_timeout_ms: Option<i64>,
   }
   
   // In resolve_multi_agent_v2_config (around line 2197):
   fn resolve_multi_agent_v2_config(config_toml: &ConfigToml) -> MultiAgentV2Config {
       let base = multi_agent_v2_toml_config(config_toml.features.as_ref());
       // ... existing resolution ...
       MultiAgentV2Config {
           // ... existing fields ...
           shutdown_timeout_ms: base
               .and_then(|b| b.shutdown_timeout_ms)
               .unwrap_or(DEFAULT_SHUTDOWN_TIMEOUT_MS),
           // ...
       }
   }

   Add validation:

   // In validate_config (around line 2999):
   validate_multi_agent_v2_wait_timeout(
       "features.multi_agent_v2.shutdown_timeout_ms",
       multi_agent_v2.shutdown_timeout_ms,
   )?;

   2.3 Alternative Approach: Abort Without Config

   If adding a config option is too invasive, the simpler approach uses a hardcoded default:

   // Minimal fix — just add timeout with a hardcoded default
   const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(30);
   
   pub(crate) async fn shutdown_live_agent(&self, agent_id: ThreadId) -> CodexResult<String> {
       let state = self.upgrade()?;
       let result = if let Ok(thread) = state.get_thread(agent_id).await {
           thread.codex.session.ensure_rollout_materialized().await;
           thread.codex.session.flush_rollout().await?;
           let result = if matches!(thread.agent_status().await, AgentStatus::Shutdown) {
               Ok(String::new())
           } else {
               state.send_op(agent_id, Op::Shutdown {}).await
           };
           // NEW: Timeout after 30s, then force abort
           if timeout(SHUTDOWN_TIMEOUT, thread.wait_until_terminated()).await.is_err() {
               tracing::warn!(thread_id = %agent_id, "shutdown timed out, force-aborting");
               thread.abort_session_loop();
           }
           result
       } else {
           state.send_op(agent_id, Op::Shutdown {}).await
       };
       let _ = state.remove_thread(&agent_id).await;
       self.state.release_spawned_thread(agent_id);
       result
   }

   ---

   3. Code Snippets

   3.1 Full shutdown_live_agent with Fix

   use std::time::Duration;
   use tokio::time::timeout;
   
   /// Submit a shutdown request for a live agent without marking it explicitly closed in
   /// persisted spawn-edge state.
   ///
   /// If the agent does not respond to the shutdown signal within the configured timeout,
   /// the session loop task is forcefully aborted. This prevents the parent agent from
   /// hanging indefinitely when a subagent is unresponsive (e.g., stuck on a network call,
   /// infinite loop, or deadlocked).
   pub(crate) async fn shutdown_live_agent(&self, agent_id: ThreadId) -> CodexResult<String> {
       let state = self.upgrade()?;
       let result = if let Ok(thread) = state.get_thread(agent_id).await {
           // Ensure rollout is persisted before shutdown
           thread.codex.session.ensure_rollout_materialized().await;
           thread.codex.session.flush_rollout().await?;
   
           let result = if matches!(thread.agent_status().await, AgentStatus::Shutdown) {
               Ok(String::new())
           } else {
               state.send_op(agent_id, Op::Shutdown {}).await
           };
   
           // Wait for graceful shutdown with timeout
           let shutdown_timeout = self.get_shutdown_timeout(&thread).await;
           let terminated_gracefully = timeout(
               shutdown_timeout,
               thread.wait_until_terminated(),
           )
           .await
           .is_ok();
   
           if !terminated_gracefully {
               // Graceful shutdown timed out — force abort the session loop
               tracing::warn!(
                   thread_id = %agent_id,
                   timeout_ms = shutdown_timeout.as_millis(),
                   "agent shutdown timed out; forcefully aborting session loop"
               );
               thread.abort_session_loop();
   
               // Give the abort a short window to take effect
               let _ = timeout(
                   Duration::from_secs(5),
                   thread.wait_until_terminated(),
               )
               .await;
           }
   
           result
       } else {
           state.send_op(agent_id, Op::Shutdown {}).await
       };
   
       let _ = state.remove_thread(&agent_id).await;
       self.state.release_spawned_thread(agent_id);
       result
   }

   3.2 CodexThread Extensions

   use std::time::Duration;
   use tokio::time::timeout;
   
   impl CodexThread {
       /// Wait until the underlying session loop has terminated, with a timeout.
       ///
       /// Returns `Ok(())` if the session loop terminated normally within the timeout,
       /// or `Err(())` if the timeout elapsed.
       pub async fn wait_until_terminated_with_timeout(
           &self,
           timeout_duration: Duration,
       ) -> Result<(), ()> {
           let termination = self.codex.session_loop_termination.clone();
           match timeout(timeout_duration, termination).await {
               Ok(()) => Ok(()),
               Err(_) => Err(()),
           }
       }
   
       /// Forcefully abort the session loop task.
       ///
       /// This is a last-resort measure when graceful shutdown has timed out.
       /// The abort will cancel any in-flight futures in the session loop,
       /// but may leave resources in an inconsistent state.
       pub fn abort_session_loop(&self) {
           if let Some(ref handle) = self.codex.session_loop_handle {
               if !handle.is_finished() {
                   handle.abort();
               }
           }
       }
   }

   3.3 Config Example

   # codex.toml
   [features.multi_agent_v2]
   # Timeout for graceful shutdown of subagents (milliseconds).
   # If a subagent does not shut down within this time, its session loop
   # is forcefully aborted. Default: 30000 (30 seconds).
   shutdown_timeout_ms = 30000
   
   # Minimum timeout for wait_agent tool calls
   min_wait_timeout_ms = 2500
   
   # Maximum timeout for wait_agent tool calls
   max_wait_timeout_ms = 120000
   
   # Default timeout for wait_agent tool calls
   default_wait_timeout_ms = 30000

   ---

   4. Testing Strategy

   4.1 Unit Tests

   Test 1: Graceful Shutdown Within Timeout

   #[tokio::test]
   async fn shutdown_live_agent_completes_gracefully_when_responsive() {
       let (mut session, turn) = make_session_and_context().await;
       let manager = thread_manager();
       let config = turn.config.as_ref().clone();
       let thread = manager.start_thread(config).await.expect("start thread");
       let agent_id = thread.thread_id;
   
       // Submit shutdown op to make the agent responsive
       thread.submit(Op::Shutdown {}).await.expect("submit shutdown");
   
       let result = manager
           .agent_control()
           .shutdown_live_agent(agent_id)
           .await;
   
       assert!(result.is_ok(), "shutdown should succeed for responsive agent");
   }

   Test 2: Timeout and Force Abort

   #[tokio::test]
   async fn shutdown_live_agent_times_out_and_force_aborts_unresponsive_agent() {
       let (mut session, turn) = make_session_and_context().await;
       let manager = thread_manager_for_unresponsive_test();
       let config = turn.config.as_ref().clone();
   
       // Create a thread that will never respond to shutdown
       let thread = manager
           .start_thread_with_blocking_session_loop(config)
           .await
           .expect("start thread");
       let agent_id = thread.thread_id;
   
       let start = Instant::now();
       let result = manager
           .agent_control()
           .shutdown_live_agent(agent_id)
           .await;
   
       let elapsed = start.elapsed();
   
       // Should complete within ~35s (30s timeout + 5s abort wait)
       assert!(
           elapsed < Duration::from_secs(60),
           "shutdown should complete within timeout, took {:?}",
           elapsed
       );
       assert!(result.is_ok(), "shutdown should succeed even after force abort");
   
       // Verify the thread was cleaned up
       assert_eq!(
           manager.agent_control().get_status(agent_id).await,
           AgentStatus::NotFound,
           "agent should be removed after force abort"
       );
   }

   Test 3: Cascade Shutdown with Mixed Responsiveness

   #[tokio::test]
   async fn shutdown_agent_tree_handles_mixed_responsiveness() {
       // Spawn parent -> child1 (responsive) -> child2 (unresponsive)
       // Closing parent should eventually complete despite child2 being unresponsive
   }

   Test 4: Config Timeout Respected

   #[tokio::test]
   async fn shutdown_uses_configured_timeout() {
       let mut config = test_config();
       config.multi_agent_v2.shutdown_timeout_ms = 5000; // 5 seconds
   
       // Verify the 5s timeout is used, not the default 30s
   }

   4.2 Integration Tests

   Test 5: Real Unresponsive Agent (API Hang Simulation)

   #[tokio::test]
   async fn close_agent_handles_api_hang_scenario() {
       // Simulate a subagent that is stuck waiting for an API response
       // by blocking the session loop model client
       // Verify close_agent completes within a reasonable time
   }

   Test 6: Multiple Concurrent Close Operations

   #[tokio::test]
   async fn close_multiple_agents_concurrently() {
       // Spawn 5 agents, make 2 unresponsive
       // Close all 5 concurrently
       // Verify all complete within timeout
   }

   4.3 Regression Tests

   Test 7: Existing Tests Still Pass

   All existing tests in multi_agents_tests.rs and control_tests.rs should continue to pass:

   just test -p codex-core -- multi_agent
   just test -p codex-core -- agent::control::tests

   4.4 Performance Tests

   Test 8: Timeout Does Not Affect Normal Shutdown

   #[tokio::test]
   async fn graceful_shutdown_latency_unchanged() {
       // Measure time for responsive agent shutdown
       // Should be < 100ms (no timeout overhead)
   }

   ---

   5. Impact Assessment

   5.1 Positive Impacts

   Area	Impact
   Reliability	Parent agents no longer hang indefinitely when closing unresponsive subagents
   Observability	Warning logs provide visibility into shutdown issues
   Resource Management	Resources are eventually cleaned up even for stuck agents
   User Experience	Users see timely feedback instead of silent hangs

   5.2 Potential Risks

   Risk	Mitigation
   Force abort may leave resources inconsistent	The abort only cancels the session loop task; cleanup (thread removal, spawn slot release) still proceeds. MCP connections are handled by shutdown_session_runtime which runs before the timeout.
   Rollout data loss	ensure_rollout_materialized() and flush_rollout() are called before the timeout, so rollout data is persisted before any force abort
   Timeout too short for slow agents	Configurable timeout allows users to adjust. Default of 30s is generous for normal operations.
   Breaking change for long-running tools	Agents with intentionally long-running operations (>30s) may be force-aborted. Users can increase the timeout.

   5.3 Backward Compatibility

   • Config: New shutdown_timeout_ms field is optional with a sensible default (30s)
   • API: No changes to public interfaces; close_agent still returns CloseAgentResult
   • Behavior: Graceful shutdown path is unchanged; timeout only activates when shutdown is slow
   • Tests: Existing tests continue to pass (test mode completes immediately)

   5.4 Files Modified

   File	Changes
   codex-rs/core/src/agent/control.rs	Add timeout to shutdown_live_agent, add get_shutdown_timeout helper
   codex-rs/core/src/codex_thread.rs	Add wait_until_terminated_with_timeout and abort_session_loop
   codex-rs/core/src/session/mod.rs	Store JoinHandle in Codex struct
   codex-rs/core/src/config/mod.rs	Add shutdown_timeout_ms to MultiAgentV2Config
   codex-rs/core/src/agent/control_tests.rs	Add tests for timeout and force abort
   codex-rs/core/src/tools/handlers/multi_agents_tests.rs	Add integration tests for unresponsive agents

   5.5 Performance Impact

   • Normal case (responsive agent): Negligible — timeout is only checked after graceful shutdown completes
   • Timeout case (unresponsive agent): Adds ~30s worst-case delay (vs. indefinite hang before)
   • Memory: Minimal — one additional Option<JoinHandle> per Codex instance

   5.6 Security Considerations

   • Force abort does not introduce new security vulnerabilities
   • The abort mechanism is the same as tokio::task::JoinHandle::abort() which is a standard Rust pattern
   • No user input is involved in the timeout decision

   ---

   6. Rollout Plan

   Phase 1: Core Fix (PR #1)

   1. Add shutdown_timeout_ms to config with default of 30s
   2. Add timeout to shutdown_live_agent
   3. Add abort_session_loop capability
   4. Add unit tests

   Phase 2: Observability (PR #2)

   1. Add metrics for shutdown timeouts
   2. Add structured logging for force aborts
   3. Add telemetry for shutdown latency distribution

   Phase 3: Tuning (PR #3)

   1. Monitor production timeout rates
   2. Adjust default if needed based on real-world data
   3. Add documentation for the new config option

   ---

   7. References

   • Issue: openai/codex multi_agent_v1.close_agent can hang for hours when closing an unresponsive subagent #24389
   • Related files:
     • codex-rs/core/src/agent/control.rs — AgentControl::shutdown_live_agent
     • codex-rs/core/src/codex_thread.rs — CodexThread::wait_until_terminated
     • codex-rs/core/src/session/mod.rs — Codex struct, SessionLoopTermination
     • codex-rs/core/src/session/handlers.rs — submission_loop
     • codex-rs/core/src/config/mod.rs — MultiAgentV2Config
   • Similar patterns in codebase:
     • app-server/src/in_process.rs — SHUTDOWN_TIMEOUT = Duration::from_secs(5)
     • app-server-client/src/remote.rs — SHUTDOWN_TIMEOUT for remote connections
5. diogovalada commented on May 28, 2026

   

   diogovalada

   on May 28, 2026

   More actions

   I hit a related long-running subagent lifecycle failure and inspected the current public source. The close_agent hang described here looks consistent with an unbounded shutdown await in the shared close path.

   Both multi_agents/close_agent.rs and multi_agents_v2/close_agent.rs call AgentControl::close_agent(...), which reaches shutdown_agent_tree(...) and then shutdown_live_agent(...).

   In codex-rs/core/src/agent/control.rs, shutdown_live_agent sends Op::Shutdown and then awaits:

   thread.wait_until_terminated().await;

   without an outer timeout or fallback. If the child session loop is stuck, blocked in a tool call, transport-stalled, or otherwise unable to process/complete shutdown, the parent close_agent tool can remain blocked indefinitely.
6. tianpeng-dev commented on Jun 14, 2026

   

   tianpeng-dev

   on Jun 14, 2026

   More actions

   I hit this issue in Codex Desktop as well and prepared a minimal branch with a bounded shutdown wait:

   https://github.com/tianpeng-dev/codex/tree/fix/bound-agent-shutdown-wait

   Commit: ae4888b fix(core): bound live agent shutdown wait

   Local evidence points to the same lifecycle path described here: shutdown_live_agent submits Op::Shutdown and then waits on thread.wait_until_terminated() without a timeout. If the child session loop is stalled, close_agent can remain blocked and the UI stays in Closing ....

   The branch keeps the close_agent result schema unchanged and adds a bounded wait around live agent termination. If the shutdown request was submitted but the agent does not terminate, it returns CodexErr::RequestTimeout and continues the existing cleanup path.

   Validated locally with:

   • cargo fmt --check -p codex-core
   • cargo test -p codex-core agent::control::legacy::tests::wait_for_live_agent_termination_times_out_when_agent_never_exits
   • cargo test -p codex-core close_agent_submits_shutdown_and_returns_previous_status
   • RUST_MIN_STACK=8388608 cargo test -p codex-core shutdown_agent_tree_closes_live_descendants

   Note: shutdown_agent_tree_closes_live_descendants overflowed the default test stack locally, but passed with RUST_MIN_STACK=8388608.

   Would the Codex team be open to this minimal approach, and if so, could you invite a PR for this branch?
7. bobashopcashier commented on Jun 17, 2026

   

   bobashopcashier

   on Jun 17, 2026

   More actions

   Fresh reproduction from June 17, 2026, on a local Codex Desktop session. This is the same failure mode: multi_agent_v1.close_agent waited until the user aborted the parent turn, and subsequent subagent work hit the thread cap.

   The affected target in both long closes was 019ed2f7-e5e2-7320-94cb-c90c6e56c428.

   2026-06-17T00:36:02.855Z  function_call         multi_agent_v1 close_agent call_xRw562qjZJKLy8HUPPcn9DAb {"target":"019ed2f7-e5e2-7320-94cb-c90c6e56c428"}
   2026-06-17T02:18:00.063Z  function_call_output                         call_xRw562qjZJKLy8HUPPcn9DAb aborted by user after 6116.8s
   2026-06-17T20:58:22.407Z  function_call_output                         call_dhhgM6qfUlsSTdHpHBoH4VxC collab spawn failed: agent thread limit reached
   2026-06-17T20:58:28.346Z  function_call         multi_agent_v1 close_agent call_Kujn6yqMZf6RASeBXYGYeasT {"target":"019ed2f7-e5e2-7320-94cb-c90c6e56c428"}
   2026-06-17T21:26:03.701Z  function_call_output                         call_Kujn6yqMZf6RASeBXYGYeasT aborted by user after 1655.3s
   2026-06-17T21:34:02.091Z  function_call_output                         call_j53aGpcWIHbyIdSlj1GvSKil collab spawn failed: agent thread limit reached
   2026-06-17T21:46:08.557Z  function_call_output                         call_K08XWC50awPzYwUGPS88eJm7 collab spawn failed: agent thread limit reached
   2026-06-17T21:47:29.771Z  function_call_output                         call_ii0uQP0YZySgYSkipxp0nWp6 collab spawn failed: agent thread limit reached
   

   I traced the current main path to codex-rs/core/src/agent/control/legacy.rs: shutdown_live_agent submits Op::Shutdown and then awaits thread.wait_until_terminated() without bounding the wait. A focused patch is forthcoming that interrupts an active agent task before queuing shutdown and wraps the termination wait in a timeout so the parent close tool cannot hang indefinitely.
8. bobashopcashier commented on Jun 17, 2026

   

   bobashopcashier

   on Jun 17, 2026

   More actions

   A focused patch is available here:

   • Fork draft PR: fix: bound multi-agent close shutdown bobashopcashier/codex#1
   • Branch: https://github.com/bobashopcashier/codex/tree/fix/multi-agent-close-hang
   • Commit: 45b61dc0a5 fix: bound multi-agent close shutdown

   I attempted to open the PR directly against openai/codex, but this account is blocked from creating upstream pull requests:

   pull request create failed: GraphQL: bobashopcashier does not have the correct permissions to execute `CreatePullRequest` (createPullRequest)
   

   The REST endpoint was also unavailable from this token:

   POST /repos/openai/codex/pulls -> HTTP 404 Not Found
   

   Net change in the branch:

   • Interrupt an active agent task before submitting Op::Shutdown.
   • Bound the live-agent termination wait with a 10 second timeout.
   • Add a regression test for shutdown_live_agent while the agent has an active turn task.

   Local verification on the branch:

   git diff --check
   cargo fmt --check
   CARGO_NET_GIT_FETCH_WITH_CLI=true cargo test -p codex-core shutdown_live_agent_closes_agent_with_active_turn
   CARGO_NET_GIT_FETCH_WITH_CLI=true RUST_MIN_STACK=8388608 cargo test -p codex-core shutdown_agent_tree
   CARGO_NET_GIT_FETCH_WITH_CLI=true RUST_MIN_STACK=8388608 cargo test -p codex-core agent::control::tests::
   
9. crazypenguin commented on Jun 20, 2026

   

   crazypenguin

   on Jun 20, 2026

   More actions

   Fresh Codex Desktop datapoint from 2026-06-20 that looks like the same close-hang family, with an extra UI/registry mismatch signal.

   Environment:

   • Codex Desktop app: 26.616.32156 (4157)
   • Bundled app CLI: codex-cli 0.142.0-alpha.1
   • Local CLI: codex-cli 0.141.0
   • Platform: macOS 15.7.7 / Darwin 24.6.0 arm64
   • Surface: Codex Desktop app, local/worktree thread with multiple subagents

   Observed sequence:

   1. A long multi-subagent worktree task auto-compacted the parent context.
   2. The parent continued after compaction and began closing a subagent named Zeno.
   3. The parent transcript stayed stuck at a progress state equivalent to "closing 1 agent" / step 4 of 5.
   4. At the same time, the Desktop right rail Subagents list no longer contained Zeno; it only showed other subagents such as Hooke, Halley, Kepler, and Turing.
   5. No visible work was running in Zeno, and there was no visible subagent card/row to inspect for it.
   6. The parent turn remained blocked waiting for the close operation.

   Why this seems useful for this issue:

   • This is not the stale-card variant where the Desktop UI still shows a closed subagent.
   • It is the opposite mismatch: the UI/listing no longer shows the target subagent, but the parent orchestrator still appears to hold a close waiter for it.
   • The likely trigger involved parent context compaction/resume, so this may intersect with Subagent completion result can be lost after parent thread compaction #26728, but the user-visible stuck point is still the close path.

   Expected behavior:

   • Closing a missing, already-detached, already-shutdown, or no-longer-listed subagent should be idempotent and complete promptly.
   • After compaction/resume, the parent's active close/wait set should reconcile with the live subagent registry/UI state.
   • A missing target should be reported as already_closed, already_shutdown, or not_found_after_spawn, not block the parent turn indefinitely.

   I am not attaching the raw screenshot/logs here because they include workspace/task details, but the key visible contradiction was:

   Parent transcript: closing subagent "Zeno" / closing 1 agent
   Desktop right rail: no "Zeno" entry in Subagents list
   Parent state: still blocked on close
   

   This supports the same fix direction discussed above: make close_agent bounded and idempotent, and make the close path treat missing/terminal/already-detached child state as a terminal cleanup result rather than waiting forever on a termination future or stale handle.
10. fujiwarakasei commented on Jun 23, 2026

    

    fujiwarakasei

    on Jun 23, 2026 – with ChatGPT Codex Connector

    More actions

    I can reproduce a closely matching case on Codex Desktop where a normal delegated message to an old parent thread completes quickly, but asking that same parent thread to close an already-unresponsive subagent leaves the parent turn in progress.

    Environment / context:

    • Codex Desktop on macOS (local host)
    • Parent thread id: 019ee0a0-c3b5-7843-951c-0b6d93258937
    • Subagent id: 019ef395-2511-7d03-9cef-4aef2fa407fd
    • Subagent role/nickname from session meta: explorer / Mendel
    • Parent cwd: /Users/fujiwarakasei/Documents/Project/LAUNOVA
    • Subagent session log: ~/.codex/sessions/2026/06/23/rollout-2026-06-23T17-24-58-019ef395-2511-7d03-9cef-4aef2fa407fd.jsonl

    Observed sequence:

    1. The subagent session log only contains session metadata, task start, and turn abort records. It produced no assistant output and no tool output.
    2. In the parent thread, earlier wait_agent calls for that subagent timed out with empty status, then send_input(... interrupt=true ...) also failed to produce a usable result.
    3. The parent attempted multi_agent_v1.close_agent for that subagent, and the user interrupted the turn while it was still stuck around the close operation.
    4. As a control, I sent a minimal delegated probe to the same parent thread: “do not use tools, just reply received”. It completed successfully in about 9.7 seconds.
    5. I then sent another delegated diagnostic prompt asking the same parent thread to close subagent 019ef395-2511-7d03-9cef-4aef2fa407fd. The parent emitted “开始关闭 sub-agent。” and then stayed inProgress; after more than two minutes there was still no close_agent return or final response.

    This makes the failure mode look specific to multi_agent_v1.close_agent on an aborted/unresponsive subagent, rather than the parent thread being generally unable to receive or answer new messages.

    The local evidence strongly matches this issue’s report: wait_agent gives no useful status for the subagent, and close_agent can block the parent turn instead of returning promptly or reporting a bounded failure.
11. safal207 commented on Jun 24, 2026

    

    safal207

    on Jun 24, 2026

    More actions

    This issue maps closely to LS work on bounded lifecycle transitions, execution leases, and durable termination receipts.

    The central distinction is:

    termination requested
    ≠ termination completed
    ≠ cleanup completed
    ≠ parent must remain blocked

    A control-plane operation such as "close_agent" should never depend indefinitely on the responsiveness of the child it is trying to terminate.

    A possible termination request:

    {
    "schema_version": "agent-termination-request/v0.1",
    "request_id": "terminate-184",
    "dispatch_id": "dispatch-review-05",
    "parent_agent_id": "agent-root",
    "child_agent_id": "agent-reviewer",
    "runtime_instance_id": "runtime-7712",
    "mode": "GRACEFUL",
    "requested_at": "2026-06-24T20:10:00Z",
    "grace_timeout_ms": 10000,
    "hard_timeout_ms": 30000
    }

    A bounded shutdown path could be:

    request graceful cancellation
    → wait within grace timeout
    → escalate to forced termination
    → persist terminal or unresolved state
    → return control to parent
    → continue asynchronous cleanup if necessary

    The key invariant would be:

    «Failure to confirm child termination must not block the parent agent indefinitely.»

    A second invariant:

    «"close_agent" should be idempotent and return the current lifecycle state when the child is already aborted, terminated, missing, or under cleanup.»

    Possible results:

    TERMINATED
    ALREADY_TERMINATED
    TERMINATION_PENDING
    FORCE_TERMINATED
    NOT_FOUND
    TERMINATION_FAILED
    CLEANUP_INCOMPLETE

    A structured result could look like:

    {
    "request_id": "terminate-184",
    "child_agent_id": "agent-reviewer",
    "status": "CLEANUP_INCOMPLETE",
    "runtime_terminated": true,
    "remaining_resources": [
    "mcp-process:xcodebuildmcp",
    "node-repl:process-889"
    ],
    "parent_blocked": false,
    "follow_up": "BACKGROUND_RECONCILIATION"
    }

    This separates three concerns that should not be collapsed:

    child task status
    child runtime status
    descendant-resource cleanup status

    For example, the child turn may already be "ABORTED", while an MCP descendant remains alive. That should not require keeping the parent turn blocked for hours.

    The parent should receive a bounded receipt:

    {
    "event_type": "AGENT_TERMINATION_RECEIPT",
    "child_agent_id": "agent-reviewer",
    "terminal_status": "ABORTED",
    "termination_confirmed": true,
    "cleanup_confirmed": false,
    "elapsed_ms": 30000,
    "decision": "RETURN_TO_PARENT"
    }

    A reconciliation worker could then asynchronously inspect:

    agent registry
    ↔ operating-system process tree
    ↔ MCP child-process registry
    ↔ launch-slot accounting

    and emit a later cleanup receipt.

    A deterministic conformance fixture could test:

    1. responsive child closes gracefully;
    2. unresponsive child reaches grace timeout;
    3. forced termination returns within the hard timeout;
    4. already-aborted child returns "ALREADY_TERMINATED";
    5. repeated close requests are idempotent;
    6. missing child does not block;
    7. leaked MCP process produces "CLEANUP_INCOMPLETE";
    8. parent regains control even when cleanup is incomplete;
    9. launch slot is released only when the relevant runtime state is reconciled;
    10. termination events remain replayable after session restart.

    Related LS work:

    • causal audit:
      Trusted Runtime: integrate CML causal audit and Cognitive Trail validation safal207/LS#594

    • execution and evidence gates:
      Trusted Runtime: add Pythia evidence gates and ProofPath authorization bundles safal207/LS#595

    • deterministic replay:
      Trusted Runtime: add LTP replay and LiminalDB event persistence adapters safal207/LS#597

    • recovered continuation state:
      feat: add recovered-evidence envelope v0.2 safal207/LS#651

    Would a small vendor-neutral "AgentTerminationRequest", "TerminationReceipt", and asynchronous cleanup fixture help make shutdown latency, escalation, idempotency, and parent non-blocking behavior machine-testable?
12. alygg77 commented on Jun 25, 2026

    

    alygg77

    on Jun 25, 2026

    More actions

    Adding a fresh matching datapoint from 2026-06-24; I posted the detailed timeline on #25870 because that issue focuses on stale/unresponsive child cleanup, but this is the same user-visible failure class.

    Short version: a native explorer subagent timed out under wait_agent three times, then multi_agent_v1.close_agent was called at 2026-06-24T04:51:56Z and did not return until the user aborted at 2026-06-24T11:56:27Z. The outputs were aborted by user after 24930.6s / 24930.5s. The child transcript had only session metadata, task_started, and turn_aborted; no assistant/tool output.

    This reinforces that close_agent needs a bounded shutdown/cleanup path and should not keep the parent turn blocked while waiting on an unresponsive child runtime.
13. alygg77 commented on Jun 25, 2026

    

    alygg77

    on Jun 25, 2026

    More actions

    Follow-up: I pushed a minimal candidate fix from this reproduction to a public fork branch:

    • branch: https://github.com/alygg77/codex/tree/codex/bound-v1-close-agent-shutdown
    • commit: ca28901
    • compare: https://github.com/openai/codex/compare/main...alygg77:codex/bound-v1-close-agent-shutdown?expand=1

    The patch bounds the v1 live-agent termination wait and returns the existing request-timeout error if the child accepts shutdown but does not terminate, while leaving the still-live thread tracked instead of removing/releasing it. I attempted to open a draft PR, but GitHub rejected both CreatePullRequest and REST pull creation for this token/repo permission set.
14. gm-fire commented on Jun 27, 2026

    

    gm-fire

    on Jun 27, 2026

    More actions

    Adding a fresh sanitized datapoint from Codex Desktop on 2026-06-27. This looks like the same multi-agent lifecycle failure family described here, with one nuance: the user-visible stuck point was a close_agent attempt on an old timed-out review subagent, while the persisted parent transcript later shows the parent turn still in progress around multi-agent wait/cleanup orchestration.

    Environment:

    • Codex Desktop on macOS 15.3.1 arm64 / Darwin 24.3.0
    • Bundled/local CLI: codex-cli 0.142.3
    • Parent session originator: Codex Desktop
    • Parent session source: vscode
    • Parent session id: 019f083a-d4d6-7b12-a4be-254b1393943f
    • Local project path and task details redacted

    Observed sequence, sanitized:

    2026-06-27T15:58:13Z spawn_agent -> review child 019f09cd-8838-76c0-bc13-874ef554a598
    2026-06-27T15:58:20Z wait_agent timeout_ms=180000
    2026-06-27T16:01:20Z wait_agent -> {"status":{},"timed_out":true}
    2026-06-27T16:01:29Z send_input interrupt=true to the review child
    2026-06-27T16:01:38Z wait_agent timeout_ms=60000
    2026-06-27T16:02:17Z wait_agent returned a completed review result
    2026-06-27T16:02:27Z close_agent review child
    2026-06-27T16:02:27Z close_agent returned previous_status promptly
    
    2026-06-27T16:06:13Z spawn_agent -> narrow follow-up review child 019f09d4-ded7-77f0-a22c-78447b67eea9
    2026-06-27T16:06:22Z wait_agent timeout_ms=120000
    2026-06-27T16:07:31Z wait_agent returned completed PASS
    2026-06-27T16:07:52Z close_agent follow-up review child
    2026-06-27T16:07:53Z close_agent returned previous_status promptly
    
    2026-06-27T16:08:35Z spawn_agent -> worker child 019f09d7-08ee-7bb3-b742-684405012763
    2026-06-27T16:08:44Z parent wait_agent timeout_ms=300000
    

    At this point the parent transcript stopped with the wait_agent call and no corresponding output in the local JSONL transcript. The Codex app still reports that parent turn as inProgress with completedAt: null.

    The child transcript for 019f09d7-08ee-7bb3-b742-684405012763 shows it continued running and applying changes after the parent wait began. It progressed at least until about 2026-06-27T16:13:07Z, including successful local command outputs, but the child transcript also has no final completed/subagent notification record. So the user-visible effect was that the parent orchestration was stuck even though child work had continued.

    Additional UI-side observation from the same parent thread summary:

    • The user asked whether it had stuck again.
    • The assistant answered that it was not project code stuck; it was an attempt to close a previous timed-out review agent where close_agent itself got stuck, and it would stop touching that old agent.
    • Later in the same thread, after avoiding that old agent, the parent was still left inProgress around subsequent subagent lifecycle management.

    Local evidence checked:

    • Parent transcript: ~/.codex/sessions/2026/06/27/rollout-2026-06-27T16-38-21-019f083a-d4d6-7b12-a4be-254b1393943f.jsonl
    • Child transcripts under ~/.codex/sessions/2026/06/28/ for the child ids above
    • read_thread for the parent still reports status inProgress, completedAt: null

    I am not attaching raw logs because they include workspace and task details. The sanitized takeaway is:

    1. Timed-out review agents make the parent orchestration fragile.
    2. close_agent can be the user-visible stuck point for an old timed-out review child.
    3. Even when later close_agent calls return promptly, the parent/child lifecycle can still fail to reconcile: a child continues doing work, but the parent wait_agent never receives a terminal result and the parent turn remains inProgress.

    Expected behavior remains the same as discussed above: child close/wait lifecycle operations should be bounded and idempotent, and parent control should return with a structured timeout/partial-cleanup result instead of leaving the parent turn open indefinitely.
15. ZenAlexa commented on Jun 27, 2026

    

    ZenAlexa

    on Jun 27, 2026

    More actions

    Adding another fresh Codex Desktop datapoint. This looks like the same close-agent lifecycle bug, but with one detail that may be useful: after a rollback/resume, the parent tried to close the same stale child again and hit the same hang a second time.

    Environment:

    • Codex Desktop app: 26.623.42026 (4514)
    • Bundled/local session CLI in the current rollout: 0.142.3
    • Parent session originator: Codex Desktop
    • Parent source: vscode
    • Platform: Darwin 25.4.0 arm64 arm
    • Parent session id: 019f0a4c-dddb-7561-951f-31015132d59f
    • Target subagent id: 019f09ce-7776-75f0-9dca-491a96971b27
    • Target subagent role/nickname: explorer / Fermat

    Observed sequence:

    target subagent rollout:
    2026-06-27T16:13:57Z turn_aborted reason=interrupted duration_ms=883237
    
    parent rollout:
    line 12805 close_agent target=019f09ce-7776-75f0-9dca-491a96971b27
    line 12807 aborted by user after 4978.7s
    
    later, after continuing/rollback:
    line 13116 close_agent target=019f09ce-7776-75f0-9dca-491a96971b27
    line 13117 aborted by user after 1889.1s
    line 13141 turn_aborted
    line 13143 thread_rolled_back num_turns=3
    

    So this was not just a slow child task. The child had already reached an interrupted/aborted state, but close_agent still put the parent turn into a blocking close path. After the user interrupted and the parent thread rolled back, the same cleanup action was attempted again and wedged the parent again.

    Why this matters:

    • Cleanup became the thing that blocked useful work.
    • Interrupting the stuck close did not leave the conversation in a clean state; the parent recorded thread_rolled_back.
    • Retrying the cleanup did not behave idempotently. It re-entered the same long wait.
    • From the user side, this feels like "closing one subagent blocks every later conversation."

    This matches the current source shape discussed above. In current main, the v1 close handler still awaits session.services.agent_control.close_agent(agent_id), and the legacy shutdown path sends Op::Shutdown and then awaits thread.wait_until_terminated() without a bounded close-specific timeout. Interrupted is also not treated as final by the public status helper, which seems relevant for stale/interrupted children that are terminal enough for cleanup but not final enough for the lifecycle machinery.

    Expected behavior:

    • Closing an already interrupted, aborted, completed, missing, or stale child should return promptly.
    • close_agent should be idempotent across rollback/resume/retry.
    • If runtime termination cannot be confirmed, the tool should return a bounded result such as shutdown_timed_out, already_interrupted, cleanup_pending, or not_found, rather than holding the parent turn open.
    • A failed close should not require the user to interrupt and risk rolling back the parent conversation.

    This feels adjacent to #25870 and #25426, but the extra signal here is the repeated close after rollback/resume: the same stale interrupted child can keep re-poisoning the parent turn until the user stops touching it.
16. parabeet commented on Jul 26, 2026

    

    parabeet

    on Jul 26, 2026 – with ChatGPT Codex Connector

    More actions

    I hit a closely related Codex Desktop failure on 2026-07-26. A subagent continued running for approximately 6.5 hours after the parent task had been wrapped up. Two separate close_agent calls each returned {"previous_status":"running"}, but shutdown did not appear to take effect immediately. A subsequent wait_agent check eventually returned not_found, confirming termination only after the repeated shutdown attempts.

    The unintended runtime exhausted the user's weekly Codex usage. This looks like more than a parent call merely blocking: agent execution and usage continued after it should have been closed.

    Expected behavior: close_agent should promptly and idempotently terminate the target and descendants, stop any further usage immediately, and return a clear terminal result. A failed or delayed shutdown must be surfaced rather than appearing successful/ambiguous.

    I am keeping the task and agent IDs out of this public issue, but they can be supplied privately to OpenAI Support for trace lookup. Please include continued usage after a failed shutdown in the investigation of this lifecycle bug.
17. ZenAlexa commented on Sep 2, 2026

    

    ZenAlexa

    on Sep 2, 2026

    More actions

    Thanks @parabeet for the 6.5-hour continuation datapoint. The repeated previous_status: running responses plus continued usage sharpen the observable contract: close should promptly and idempotently terminate the target and descendants, and a failed termination should return a clear terminal result.