Skip to content

fix: decode exec and clone output once per stream so split UTF-8 survives - #1281

Merged
SaulMoro merged 1 commit into
Tencent:mainfrom
arnavsingh0418:fix/exec-clone-utf8-chunks
Oct 22, 2026
Merged

SaulMoro merged 1 commit into
Tencent:mainfrom
arnavsingh0418:fix/exec-clone-utf8-chunks

Conversation

@arnavsingh0418

Copy link
Copy Markdown
Contributor

Summary

execCommand (src/utils/exec.ts) and runCommand (src/clone.ts) decoded every stdout/stderr chunk on its own with chunk.toString(), so a multi-byte UTF-8 character split across two pipe reads became U+FFFD in both the captured string and the stream output. Both now keep one StringDecoder('utf8') per stream and flush it on close, the same approach as src/workflow/local-tool-adapter.ts:349 and #1275.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature causing existing behavior to change)
  • Documentation only
  • Refactor / internal cleanup

Test Plan

  • npx tsc --noEmit passes
  • npm run lint passes
  • npx vitest run src/__tests__/exec.test.ts src/__tests__/clone.test.ts passes (7 tests)
  • Added/updated tests for the change

New tests split a character across two chunks (exec: stdout, stderr, and the stream write path; clone: a branch name from rev-parse --abbrev-ref). On main they fail with \ufffd in the output; with this change they pass.

Real-CLI verification (npm run build, then the built CLI with a stand-in OpenCode 2 on PATH whose session list --format json writes its JSON in two pipe writes split inside 会, in a sandboxed HOME/XDG_DATA_HOME). teamai session migrate ses_utf8 -s opencode -t claude-code --cwd <ws> hits the ambiguous-prefix path, which prints the listed titles:

Before (main):

Ambiguous session id "ses_utf8": matches 2 sessions.
  ses_utf8_a  修复���话导出 café 😀
  ses_utf8_b  重构登录流程 naïve 🚀

After:

Ambiguous session id "ses_utf8": matches 2 sessions.
  ses_utf8_a  修复会话导出 café 😀
  ses_utf8_b  重构登录流程 naïve 🚀

Related Issues

Fixes #1279

Notes for Reviewers

  • Kept clone.ts in the same change so the two copies do not diverge, as the issue asked.
  • The stream path writes the decoder's remainder on close, so streamed output matches the captured string.
  • captureTail in exec.ts decodes per chunk too, but it only keeps a whitespace-collapsed tail for one log line, so I left it out to keep this change surgical. Happy to include it if you'd like.
  • I ran the full npx vitest run on Windows; failures there are timeouts and symlink cases (local-agent, workflow-cache, rules, hook-handlers, …) in files that don't import exec.ts or clone.ts. Relying on CI (Linux) for the full suite.
  • Real OpenCode / Codex installs were not exercised; the stand-in reproduces the pipe split the issue describes.

@github-actions

Copy link
Copy Markdown

No findings in the scoped PR diff.

The description documents sufficient testing: type checks, lint, focused unit tests, and a representative built-CLI run reproducing the split UTF-8 issue with before/after output. The stand-in OpenCode verification satisfies the real-CLI requirement; a full provider × agent matrix is not required.

Review was read-only. I did not run, build, or install anything from the PR.

@SaulMoro SaulMoro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during triage.

@arnavsingh0418 Approving. We built this head and main and ran teamai session migrate ses_utf8 -s opencode -t claude-code against a stand-in opencode that splits a multi-byte character across two writes. That path reaches execCommand through the OpenCode adapter's session list.

  • main: ses_utf8_a 修复���话导出 café 😀
  • this head: ses_utf8_a 修复会话导出 café 😀

The three new unit tests fail on main for the stated reason and pass here.

[P2 non-blocking] captureTail in src/utils/exec.ts:118 still decodes each chunk with chunk.toString('utf8'), so the tail it returns for an error line can carry the same replacement characters. The description says it was left out. It affects one log line, so it can be a follow-up.

The src/clone.ts change is covered by the mocked unit test only. We didn't run a real clone through it.

#1208 also edits src/clone.ts, next to this hunk. This PR owns the decoder lines and #1208 owns the token redaction in the timeout message. The two merge cleanly.

@SaulMoro
SaulMoro merged commit 8a94438 into Tencent:main Oct 22, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] exec and clone decode each pipe chunk, so a split UTF-8 character is corrupted

3 participants