Skip to content

fix(workflow): preserve UTF-8 across tool output chunks - #1242

Merged
jeff-r2026 merged 1 commit into
Tencent:mainfrom
ydflow:fix/workflow-utf8-output
Oct 21, 2026
Merged

jeff-r2026 merged 1 commit into
Tencent:mainfrom
ydflow:fix/workflow-utf8-output

Conversation

@ydflow

@ydflow ydflow commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Summary

The local workflow tool adapter decodes every stdout Buffer independently. If a UTF-8 character crosses pipe chunks, a successful Agent result can silently acquire replacement characters, which are then persisted in the run state and fallback result.json. For example, a streamed 检查完成 café 😀 can become ���查完成 café 😀 even though the workflow reports success.

Use one StringDecoder per tool launch and flush it when the process closes. JSON parsing, result precedence, metadata extraction and exit handling retain their current contracts.

Type of Change

  • Bug fix

Test Plan

Windows, Node v24.19.0, upstream baseline 8487c34d1013f52a232199492a489fea63217de2:

  • npm run build: passed.
  • npm run typecheck: passed.
  • npm run lint: passed.
  • git diff --check: passed.
  • npx vitest run src/__tests__/workflow-local-tool-adapter.test.ts -t 'local tool adapter|interactive tool invocations|reports an interactive tool exit': 17 passed, 7 skipped. The new regression feeds the JSON stream one byte per chunk, covering 2-, 3- and 4-byte UTF-8 characters, and checks the returned output and persisted fallback. It fails on the unmodified implementation with corrupted text.
  • After building, npx vitest run --config vitest.e2e.config.ts src/__tests__/e2e/workflow-utf8-output.test.ts: 1 passed. A temporary Git workspace and validated workflow cache drive the real dist/index.js workflow start unicode --driver local --tools default=claude. A fake Claude executable writes a JSON result in two pipe writes, splitting inside a Chinese character. The workflow succeeds, and both its stored step outputs and fallback result file retain the exact original string. No real model or external provider is used. The same E2E test on a pristine built baseline fails: the workflow succeeds but its stored summary is ���查完成 café 😀.

The entire adapter file was also run: 17 passed, 7 failed. All seven failures are the existing SIGTERM/SIGHUP signal expectations on Windows. A pristine upstream run reproduces the same seven failures (16 passed, 7 failed, without the new test). Full-suite green locally is not claimed; Linux/macOS signal coverage is left to CI.

Related Issues

No matching open PR or issue found in the current open-PR list or targeted UTF-8 stdout, stdout unicode, and unicode searches.

Notes for Reviewers

The production diff is confined to stdout decoding in src/workflow/local-tool-adapter.ts; other changes are tests. No CLI flags or usage contract change. The decoder is local to each launch, so parallel subprocesses do not share buffered bytes.

#1175 also edits the adapter file, but its current diff changes resume arguments and the launch input interface, rather than the stdout reader. This fix is independent of that feature. It has no file overlap or dependency on #1241.

Prepared with AI assistance; the regression and CLI verification above were executed locally.

@github-actions

Copy link
Copy Markdown

No findings in the specified diff.

The PR description documents sufficient testing: a build followed by a representative real-CLI E2E run verifying preserved UTF-8 in stored outputs and fallback result.json. It also reports baseline reproduction and clearly identifies existing Windows signal-test failures.

Review was read-only; I did not run, build, or install anything from the PR.

@jeff-r2026
jeff-r2026 merged commit 75396ce into Tencent:main Oct 21, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants