Repository navigation
Run tests/misc on the UTF-8 build in CI - #885
Open
yutaro-sakamoto wants to merge 12 commits into
Open
yutaro-sakamoto wants to merge 12 commits into
yutaro-sakamoto wants to merge 12 commits into
Conversation
codegen.c kept the inside_check/inside_stack bookkeeping from the original C code generator, but only the decrements survived the rewrite to Java code generation: nothing ever increments inside_check. On non-GCC builds (the MSVC-built cobj.exe) the counter goes negative as soon as a runtime check is attached to a reference (-debug with a variable subscript, reference modification or SEARCH ALL), and inside_stack[inside_check - 1] then writes out of bounds, crashing the compiler. GCC builds never compile these blocks, which is why Linux was unaffected. Remove the machinery entirely: the generated Java must not depend on the compiler that built cobj. Un-skip the Windows tests that were failing due to this crash. Fixes opensourcecobol#829, opensourcecobol#831, opensourcecobol#832, opensourcecobol#833, opensourcecobol#834 (sub-issues of opensourcecobol#828)
With -java-package the jar arguments were assembled as package_dir + file_path_delimitor + class, mixing the '/' separators from the package path with '\' on Windows (e.g. com/abc\prog.class), which produces malformed entry names in the archive. The jar tool accepts '/' everywhere, so join the paths with '/' unconditionally. The class file cleanup after archiving had the opposite problem: cmd.exe's del does not accept '/' separators. Build the remove paths with the native separator, let cd handle the output directory, and drop a leftover '#aaa' token that del would treat as an argument. Un-skip the Windows jar test. Fixes opensourcecobol#830 (sub-issue of opensourcecobol#828)
Removing the inside_check block exposed a variableScope finding for 'code' in joutput_stmt; declare it in the handler block where it is used.
Every program in a translation unit becomes its own .java/.class file named after the program ID, so two programs whose IDs differ only in letter case overwrite each other's files on a case-insensitive filesystem and the result silently misbehaves. On Windows, detect the collision before code generation and fail with an explicit error. The two affected tests stay skipped on Windows because the behavior they exercise is impossible there by design; a new Windows-only test verifies the diagnostic instead. Fixes opensourcecobol#835 (sub-issue of opensourcecobol#828)
On case-insensitive file systems programs whose PROGRAM-IDs differ only in letter case would generate class files with the same name; cobj now reports a compile error there, so document the restriction in the usage section.
* Use "cd /d" on Windows: cmd.exe's plain "cd" does not switch drives, so with -o pointing at another drive the jar and del commands would run against the wrong directory * Replace the unbounded strcpy into package_dir_native with snprintf and give package_name_to_path an explicit output bound, so an overlong -java-package argument can no longer overflow the buffers
utf8_calc_sjis_size() and utf8_calc_sjis_column() advance the scan pointer by COB_U8BYTE_1(), which yields 0 for anything that is not a UTF-8 lead byte. The preprocessor calls the former for every source line holding a byte >= 0x80, so a source that is not valid UTF-8 -- a Shift_JIS or EUC-JP file, a Latin-1 comment, a truncated UTF-8 file -- stops the scan dead and cobj loops forever instead of reporting anything. Both functions now consume a single byte when the lead byte is invalid or the sequence is cut short by the end of the buffer, so the scan always advances. The same bound closes an out-of-bounds read in utf8_hankaku_kana(), which looks at p[1] and p[2] without knowing whether they are still inside the buffer. This only affects --enable-utf8 builds; the Shift_JIS build reaches these functions through sjis_strlen()/sjis_pick(), which always step one byte. A source whose non-UTF-8 bytes sit past column 72, where fixed format ignores them, now compiles as it should rather than hanging the compiler.
joutput_initialize_one() parked the segment sizes of an '&' concatenated
literal in the global sgmt_sizes for joutput_string() to pick up later, but
the CB_CATEGORY_NATIONAL branch returns without emitting that literal. The
sizes stayed behind and were applied to the next, unrelated literal:
01 N1 PIC N(2) VALUE N"..." & N"...".
01 N2 PIC X(11) VALUE "a...".
N1, the literal the sizes belonged to, came out in one piece while N2 was
split at N1's offsets. On a Shift_JIS build those offsets land between the
lead and trail byte of a multibyte character, so the generated .java is no
longer valid Shift_JIS and javac rejects it with "unmappable character".
The sizes now travel with the literal into joutput_string_segments(), so a
literal is split by its own segment list and by nothing else. The cache
records the segment count as well, so walking the list cannot run off the
end of the array.
Splitting also works on a UTF-8 build now; joutput_string_write() used to
ignore the segment list there entirely.
joutput_string() and joutput_string_segments() read as if they wrote the string, but what lands in the output at that point is the name of the constant holding it; the string itself is written much later, once per cached literal, by what was called joutput_string_write(). The file already distinguishes the two elsewhere -- joutput_sql_field_ref() against joutput_alphabet_name_definition() -- so follow that: joutput_string -> joutput_string_ref joutput_string_segments -> joutput_string_ref_segmented joutput_string_write -> joutput_string_literal_definition joutput_string_segment_break() was named after a parameter rather than after its job, and it only ever helps the definition side, so it becomes joutput_string_literal_break(). Comments updated to say which half of the pair each function belongs to. No behaviour change.
misc runs on both the Shift_JIS and the UTF-8 build, so a test group whose COBOL source carries Japanese cannot live there: it only compiles under the encoding it was written in. That is why misc was commented out of the --enable-utf8 matrix. With the scan no longer hanging on those sources, the groups can be moved next to their peers -- i18n_sjis for the Shift_JIS build, cobol_utf8 for the UTF-8 one -- leaving misc encoding independent and letting it run on both. compare-national, compare-national-diff-size, japanese-char-section-var and file-handler-japanese move whole. fd-external and java-interface keep their alphanumeric groups in misc and hand over only "FD EXTERNAL nihongo" and "Japanese". convert-string-concat splits along the same line: misc keeps the PIC X half, the PIC N half goes to the i18n suites. copy-comments moves too, still commented out in the suite file as it was in misc.at. cobol_utf8's copy of java-interface no longer repeats the eleven alphanumeric groups misc already covers. java-interface asked for Shift_JIS output with -Dfile.encoding, which stopped reaching System.out in JDK 18 (JEP 400). It now also passes -Dstdout.encoding, ignored by the JDK 11 CI uses and honoured by newer ones.
yutaro-sakamoto
marked this pull request as ready for review
August 14, 2026 13:21
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
このブランチは #862 をマージしたうえで作業しています。#862 の修正
(cppcheck 指摘の対応と MSVC ビルド修正) が無いと develop 上で CI が通らない
ためです。#862 がマージされたら develop に追従して解消します。
レビュー時は先頭のマージコミット以降の4コミットをご覧ください。
概要
UTF-8 ビルド (
--enable-utf8) の CI マトリクスからtests/miscが外されていた(
#- "misc") 原因を調査し、実行できるようにしました。原因
cobj が無限ループでハングします。
cobj/cobj.cのutf8_calc_sjis_size()とutf8_calc_sjis_column()はCOB_U8BYTE_1()の返り値だけポインタを進めますが、これは UTF-8 の先頭バイトでないと 0 を返します。前処理器は
0x80以上のバイトを含むすべてのソース行でこれを呼ぶため、
misc.src/の Shift_JIS で書かれたテストソースを食わせるとポインタが進まず永久にループします。
最小再現 (UTF-8 ビルド):
→
cobjが戻ってきません (timeoutで rc=124)。テストがタイムアウトせず回り続けるため、CI ジョブごと詰まります。
Shift_JIS 限定ではなく、EUC-JP、Latin-1、壊れた UTF-8 などUTF-8 として不正な
バイト列全般で起きます (
0x80–0xBFか0xFE/0xFFが先頭位置に来たとき)。既定 (Shift_JIS) ビルドは
sjis_strlen()/sjis_pick()を通り必ず1バイト進むので影響ありません。
変更内容
1. 走査が必ず前進するようにする (
cobj/cobj.c)先頭バイトが不正な場合、またはバッファ末尾でシーケンスが切れている場合は
1バイトだけ消費します。不正なソースはハングではなくエラーとして報告されます。
utf8_hankaku_kana()がバッファ外を読み得た点も同時に塞いでいます。固定形式で72桁より後ろ（無視される領域）に非 UTF-8 の注記があるソースは、
これまでコンパイラが停止していましたが、正常にコンパイルできるようになります。
2.
&連結リテラルのセグメントサイズをリテラルと一緒に運ぶ (cobj/codegen.c)joutput_initialize_one()はセグメントサイズをグローバルsgmt_sizesに置いていましたが、
CB_CATEGORY_NATIONALの分岐はリテラルを出力せずに return するため、サイズが次の無関係なリテラルに適用されていました。
現行 develop (Shift_JIS ビルド) で、本来分割されるべき N1 が1本のまま出力され、
代わりに N2 が多バイト文字の途中で分割されます。生成された .java が壊れた
Shift_JIS になり、javac が
unmappable characterで落ちます。サイズをリテラル自身と一緒に
joutput_string_ref_segmented()へ渡すようにしました。副次的に UTF-8 ビルドでも
&分割が出るようになり (従来は未実装)、キャッシュにセグメント数を持たせたことで配列外参照も塞がれます。
3. 文字列リテラル出力関数の改名 (
cobj/codegen.c)2の作業で、これらの関数が「出力するもの」と名前が食い違っていることが
はっきりしたので改名しました。動作は変えていません。
joutput_stringjoutput_string_refjoutput_string_segmentsjoutput_string_ref_segmentedjoutput_string_writejoutput_string_literal_definitionjoutput_string_segment_breakjoutput_string_literal_break同ファイルの
joutput_sql_field_ref/joutput_alphabet_name_definitionの語彙に合わせています。
4. 文字コード依存のテストを
miscから移動 (tests/, ワークフロー)miscは両ビルドで走るため、COBOL ソースに日本語を含むテストグループはそもそも
miscに置けません。Shift_JIS ビルド用のi18n_sjisと UTF-8 ビルド用のcobol_utf8へ移し、miscを文字コード非依存にしました。両スイートとも文字コードを理由としたスキップはありません。
cobol_utf8側のjava-interfaceがmiscと重複して持っていた英数字11グループも削除しました。UTF-8 でも
miscが走るようになったのでカバレッジは維持されます。また
java-interfaceは-Dfile.encoding=SJISでSystem.outの文字コードを制御しようとしていましたが、JEP 400 (JDK 18) 以降これは効きません。
-Dstdout.encoding=SJISを併記しました (CI の JDK 11 では無視され、新しい JDK では有効)。検証
残る4件のスキップはいずれも元から
i18n_sjis.src/cobol_utf8.srcにあった${SKIP_TEST}で、本 PR とは無関係です。いずれもCFLAGS=-Werrorの CI 相当ビルドで確認しています。
Note
This branch merges #862. Without its fixes (a cppcheck finding and an MSVC
build failure) CI does not pass on develop. Once #862 lands this resolves by
following develop. For review, the four commits after the leading merge are
the ones belonging to this change.
Summary
Investigated why
tests/miscwas excluded from the--enable-utf8CI matrix(
#- "misc") and made it runnable.Root cause
cobj hangs in an infinite loop.
utf8_calc_sjis_size()andutf8_calc_sjis_column()incobj/cobj.cadvancethe scan pointer by
COB_U8BYTE_1(), which yields 0 for anything that isnot a UTF-8 lead byte. The preprocessor calls the former for every source line
containing a byte >= 0x80, which is exactly what the Shift_JIS test sources
under
misc.src/are made of, so the scan stops making progress.Minimal reproduction on a UTF-8 build:
cobjnever returns (exit 124 undertimeout). The test never times outeither, so the whole CI job stalls.
This is not specific to Shift_JIS: EUC-JP, Latin-1 and truncated UTF-8 trigger
it too — any byte in
0x80–0xBFor0xFE/0xFFreaching a lead position.The Shift_JIS build is unaffected; it goes through
sjis_strlen()/sjis_pick(),which always step one byte.
Changes
1. Keep the scan moving (
cobj/cobj.c)Consume a single byte when the lead byte is invalid or the sequence is cut
short by the end of the buffer. A malformed source is now reported instead of
hanging the compiler. This also closes an out-of-bounds read in
utf8_hankaku_kana().A source whose non-UTF-8 bytes sit past column 72, where fixed format ignores
them, now compiles as it should rather than hanging the compiler.
2. Segment sizes travel with the literal (
cobj/codegen.c)joutput_initialize_one()parked them in the globalsgmt_sizes, but theCB_CATEGORY_NATIONALbranch returns without emitting the literal, so thesizes were applied to the next, unrelated literal:
On current develop (Shift_JIS build) N1 comes out in one piece while N2 is
split in the middle of a multibyte character, leaving a .java that is no
longer valid Shift_JIS; javac rejects it with
unmappable character.The sizes now travel with the literal into
joutput_string_ref_segmented().That also lets the UTF-8 build split
&concatenations, which it never did,and the cache records the segment count so the walk cannot run off the array.
3. Renaming the string literal emitters (
cobj/codegen.c)Working on (2) made it plain that these functions are named after something
other than what they emit. No behaviour change.
joutput_stringjoutput_string_refjoutput_string_segmentsjoutput_string_ref_segmentedjoutput_string_writejoutput_string_literal_definitionjoutput_string_segment_breakjoutput_string_literal_breakFollowing
joutput_sql_field_ref/joutput_alphabet_name_definitionin thesame file.
4. Moving the encoding-dependent test groups (
tests/, workflows)miscruns on both builds, so a group whose COBOL source carries Japanesecannot live there. Those groups now sit in
i18n_sjisfor the Shift_JIS buildand
cobol_utf8for the UTF-8 one, leavingmiscencoding independent.Neither suite skips anything on account of its encoding.
cobol_utf8's copy ofjava-interfaceno longer repeats the elevenalphanumeric groups
miscalready covers — coverage is kept becausemiscnow runs on the UTF-8 build too.
java-interfacealso asked for Shift_JIS output with-Dfile.encoding, whichstopped reaching
System.outin JDK 18 (JEP 400). It now passes-Dstdout.encodingas well, ignored by the JDK 11 CI uses and honoured bynewer ones.
Verification
The four remaining skips are pre-existing
${SKIP_TEST}markers ini18n_sjis.src/cobol_utf8.srcand are unrelated to this change. All on aCI-equivalent
CFLAGS=-Werrorbuild.