Fix undefined behavior in RLE encode/decode size header handling

[vimac]

load() returns uint8_t, which promotes to int; load() << byte * 8 performs a shift of >= 32 bits on a 32-bit int when byte >= 4, which is undefined behavior. On the encode side, input.size() >> byte * 8 has the same problem, since input.size() is a 32-bit uint.

At -O2/-O3 (the performance build profile), Apple Clang miscompiles nall::Decode::RLE based on this UB and emits a brk trap at runtime: saving states appears to work, but loading a state — or even just selecting one in the State Manager, which RLE-decodes the preview image — crashes bsnes. -O1/-Og builds happen to generate working code, which made this look like a macOS-specific regression (see #396 #296).

This was reproduced independently of the UI: decoding a valid .bst state file with the exact updateSelection() preview code crashes at -O2/-O3 and works at -O1/-Os; casting to uint64_t before shifting fixes all optimization levels.

Cast to uint64_t before shifting so all eight size-header bytes are assembled and written correctly at any optimization level.

Fixes #396 #296

[Screwtapello]

Thank you very much for looking into this! I'll try to review and merge this soon.

[Screwtapello]

Building latest version on Linux with g++ 16: seems to work fine, as expected
Building latest version with clang++ 21: crash when loading save state
Building this patch with g++ 16: still works fine
Building this patch with clang++ 21: now works!

Thank you for your contribution!