Skip to content

add timeout to datastream clear_buffer to avoid deadlock when missing playback finished event - #3917

Merged
longcw merged 3 commits into
mainfrom
longc/datastream-io-clear-buffer-timeout
Nov 14, 2025
Merged

longcw merged 3 commits into
mainfrom
longc/datastream-io-clear-buffer-timeout

Conversation

@longcw

@longcw longcw commented Nov 13, 2025 •

Copy link
Copy Markdown
Contributor

when playback finished event is not received from the avatar service, the agent may hang forever to wait for the event. this usually happens during interruption due to some race conditions in the avatar service or network issues.

fix #3237 and #3434

@longcw
longcw requested a review from a team November 13, 2025 07:21

@davidzhao davidzhao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

timeout, _on_timeout
)

def _handle_playback_finished(self, data: rtc.RpcInvocationData) -> str:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QQ: Could there be a race condition here? If this function gets called after await self._room.local_participant.perform_rpc completes but before self._clear_buffer_timeout_handler = asyncio.get_event_loop().call_later( executes, would that trigger multiple on_playback_finished callbacks?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch! it could be if the playback finished received before clear buffer rpc done... will add a check in the timeout callback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checked the code, there is a check in on_playback_finished to ignore the redundant events, and ideally this shouldn't happen if avatar service handles the clear_buffer RPC before sending the playback finished event.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Thanks!

@chenghao-mou chenghao-mou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

timeout, _on_timeout
)

def _handle_playback_finished(self, data: rtc.RpcInvocationData) -> str:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Thanks!

@longcw
longcw merged commit 76f0cab into main Nov 14, 2025
17 of 18 checks passed
@longcw
longcw deleted the longc/datastream-io-clear-buffer-timeout branch November 14, 2025 00:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

When using an Avatar + Gemini Realtime = Error when interrupting

4 participants