Skip to content

std.base64 fails on unicode strings with codepoints > 255 (emoji, CJK, etc.) #887

Description

@He-Pin

Bug Description

std.base64() fails when the input string contains characters with codepoints greater than 255, such as emoji, CJK characters, and other non-Latin scripts.

Reproduction

std.base64("🎉")

Expected Behavior

"8J+OiQ=="

(This is the correct base64 encoding of the UTF-8 bytes f0 9f 8e 89 for 🎉)

Actual Behavior

RUNTIME ERROR: base64 encountered invalid codepoint value in the array (must be 0 <= X <= 255), got 127881

More Examples

std.base64("你好")    // fails - Chinese characters
std.base64("مرحبا")   // fails - Arabic characters
std.base64("héllo")   // works - all codepoints <= 255
std.base64("hello")   // works - ASCII

Version

$ jsonnet --version
Jsonnet commandline interpreter (Go implementation) v0.22.0

Analysis

The implementation appears to convert the string to an array of Unicode codepoints and then try to treat each codepoint as a byte. This fails for any character with a codepoint > 255.

The correct behavior should be:

  1. Encode the string as UTF-8 bytes
  2. Base64-encode the resulting byte array

sjsonnet handles this correctly:

std.base64("🎉")  // returns "8J+OiQ==" (correct)
std.base64("你好") // returns "5L2g5aW9" (correct)

Verification that the expected output is correct:

echo -n "🎉" | base64
# Output: 8J+OiQ==

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions