Repository navigation
Bound persistence file names in UTF-8 bytes as well as characters [patch] - #101
Merged
Merged
Conversation
…tch] GetSafeFileName truncated only past 100 UTF-16 characters, but Linux and macOS limit a file-name component to 255 bytes. A key of 83+ CJK characters stayed untruncated, and StoreAsync threw PathTooLongException once the extension and ".tmp" were appended. Names are now also truncated past 246 UTF-8 bytes, which leaves room for a five-byte extension and ".tmp". Every name that could be written before stays verbatim, so existing files are still found. Truncation takes whole code points, so a surrogate pair is no longer split, and IsTruncatedName recognises names cut at the byte budget. Fixes #48 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D7wytU6jsZ1cH3f6grCTz5
|
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #48
Problem
PersistenceProviderUtilities.GetSafeFileNametruncated a name only when it exceeded 100 UTF-16 characters. Linux and macOS limit a file-name component to 255 bytes. A key of 83 or more CJK characters was left untruncated. OnceFileSystemPersistenceProviderappended the extension and.tmp,StoreAsyncthrewPathTooLongException. Afterwards the key read back as missing. DataHome, ConfigHome and Temp delegate to the same provider, so they failed the same way.Change
.tmp, and every built-in serializer's extension (.json,.yaml,.toml) is five bytes.Truncatetakes whole code points within both budgets, so it no longer splits a surrogate pair. The old fixed 83-character cut left a lone high surrogate in an emoji key. The percent-escape back-off is unchanged.IsTruncatedNamealso recognises names cut at the byte budget: the marker plus hash, with a UTF-8 length within 5 bytes of it. Names truncated under the old rule, at 98–100 characters, are still recognised.Tests (
PersistenceNamingTests)SafeFileName_Bounds_Utf8_Bytes_Without_Splitting_A_Code_Pointruns five CJK, emoji and mixed keys. For each it checks that the name is ≤ 246 bytes, has no split surrogate, is reported as unrecoverable, and stays distinct.Long_Multibyte_Keys_Store_And_Retrieve_On_The_Native_File_Systemstores and retrieves the same five keys on the real file system.SafeFileName_Keeps_Names_That_Fit_The_Byte_Budget_Verbatimchecks that an 82-CJK name (246 bytes) is unchanged.PersistenceProviderUtilities.cs, 5 cases fail: both CJK keys in the bounds test, the 60-emoji key's split surrogate, and both CJK store/retrieve cases withPathTooLongException. All pass with the fix.Essentialsis clean on all targets, netstandard2.1 included.This PR is independent of #99 and #100 and merges cleanly with both.
🤖 Generated with Claude Code
https://claude.ai/code/session_01D7wytU6jsZ1cH3f6grCTz5
Generated by Claude Code