Skip to content

std.base64 fails on unicode strings with codepoints > 255 (emoji, CJK, etc.) #887

Description

@He-Pin

Bug Description

std.base64() fails when the input string contains characters with codepoints greater than 255, such as emoji, CJK characters, and other non-Latin scripts.

Reproduction

std.base64("🎉")

Expected Behavior

"8J+OiQ=="

(This is the correct base64 encoding of the UTF-8 bytes f0 9f 8e 89 for 🎉)

Actual Behavior

RUNTIME ERROR: base64 encountered invalid codepoint value in the array (must be 0 <= X <= 255), got 127881

More Examples

std.base64("你好")    // fails - Chinese characters
std.base64("مرحبا")   // fails - Arabic characters
std.base64("héllo")   // works - all codepoints <= 255
std.base64("hello")   // works - ASCII

Version

$ jsonnet --version
Jsonnet commandline interpreter (Go implementation) v0.22.0

Analysis

The implementation appears to convert the string to an array of Unicode codepoints and then try to treat each codepoint as a byte. This fails for any character with a codepoint > 255.

The correct behavior should be:

  1. Encode the string as UTF-8 bytes
  2. Base64-encode the resulting byte array

sjsonnet handles this correctly:

std.base64("🎉")  // returns "8J+OiQ==" (correct)
std.base64("你好") // returns "5L2g5aW9" (correct)

Verification that the expected output is correct:

echo -n "🎉" | base64
# Output: 8J+OiQ==

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions