COBOL: full-width hyphens (－ U+FF0D / − U+2212) and full-width digits still truncate Japanese data names and drop sections (#3955 follow-up) #3991

Follow-up to #3955 (fixed by #3967). Found while rescanning the opensourcecobol4j fresh estate (#3806, epic #3803) on origin/main 6991970d. This was a pre-trial scan with no porting.

Defect

#3967 lets COBOL words contain Unicode letters. Japanese COBOL words also contain two other kinds of character that the name classes still reject:

• Full-width hyphens: － U+FF0D, which cp932 decodes byte 0x817C to, and − U+2212, which Python's shift_jis decodes the same byte to. COBOL: Japanese user-defined words are invisible -- a Japanese PROGRAM-ID drops the program from refraction; data names truncated or missing; sections missing #3955 named both.
• Full-width digits ０-９ (U+FF10-FF19).

What happens:

• Data names are truncated at the first such character. Different items can then collide: ＴＥＳＴ−ＤＡＴＡ１ and ＴＥＳＴ−ＲＥＣＯＲＤ１ both become ＴＥＳＴ.
• Sections are dropped entirely. 主処理 SECTION is found, but Ｓ−初期化 SECTION, Ｓ－終了 SECTION and ｓ１２３ SECTION are not in function_data at all.

On the estate (independent Unicode-aware reading vs the master DB)

Every remaining truncation is at a full-width hyphen or digit: Ｏ−文字列 -> Ｏ, 項目２ -> 項目, ＴＥＳＴ−ＤＡＴＡ１ -> ＴＥＳＴ, 横浜−１ -> 横浜, ｗｋ－０３ -> ＷＫ, 項目ＡＢＣＤＥＦＧＨ０１２３４5 -> 項目ＡＢＣＤＥＦＧＨ.

(The missing-data-name count went to 0 and the Japanese PROGRAM-IDs are all refracted now. #3967 did fix the first half.)

Minimal reproduction (origin/main 6991970)

mkdir -p estate
cat > estate/JPWORDS.cbl <<'COB'
       IDENTIFICATION DIVISION.
       PROGRAM-ID. JPWORDS.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01  項目２           PIC X(4).
       01  Ｏ−文字列       PIC X(4).
       01  ＷＫ－０３       PIC X(4).
       01  横浜-1           PIC X(4).
       PROCEDURE DIVISION.
       主処理 SECTION.
           MOVE 'A' TO 項目２.
       ASCII-SEC SECTION.
           DISPLAY 'X'.
       Ｓ−初期化 SECTION.
           DISPLAY 'Y'.
       ｓ１２３ SECTION.
           DISPLAY 'Z'.
       Ｓ－終了 SECTION.
           STOP RUN.
COB
GITGALAXY_DISABLE_GIT_HISTORY=1 python -m gitgalaxy.galaxyscope estate --db-only --output out
sqlite3 out/estate_galaxy_master.db "select item_name from record_data; select func_name from function_data"

Actual:

項目
Ｏ
ＷＫ
横浜-1
ASCII-SEC
主処理

Expected: 項目２, Ｏ−文字列, ＷＫ－０３, 横浜-1, plus all five sections.

Notes

• Treat － / − as the COBOL hyphen inside a word (the same rules as -: not first, not last). Accept full-width digits wherever ASCII digits are accepted. ID_CONTINUE already covers digits (Nd), so the COBOL class probably just needs to use it rather than ID_START alone after the first character.
• Case folding looks right: ｗｋ is stored as ＷＫ.
• A test should cover both SJIS decodings of 0x817C (U+2212 via shift_jis, U+FF0D via cp932).

🤖 Generated with Claude Code