signed left-shift overflow in vorbis_book_init_decode

[iceray00]

Summary

Malformed Vorbis input can trigger signed left-shift undefined behavior during decode setup in vorbis_book_init_decode(). The failing expression is in lib/sharedbook.c, where an int loop index is left-shifted into the sign range while constructing the first-level decode table:

sharedbook.c:425:30: runtime error: left shift of 16 by 27 places cannot be represented in type 'int'

I reproduced this with a standalone libFuzzer harness based on contrib/oss-fuzz/decode_fuzzer.cc. The tested build identifies as Vorbis v1.3.7. The input is parsed through the normal vorbisfile decode flow and reaches ov_read(), so applications decoding untrusted Vorbis streams can hit this as a deterministic sanitizer abort.

Details

The relevant code in lib/sharedbook.c is:

ogg_uint32_t mask=0xfffffffeUL<<(31-c->dec_firsttablen);
long lo=0,hi=0;

for(i=0;i<tabn;i++){
  ogg_uint32_t word=i<<(32-c->dec_firsttablen);
  if(c->dec_firsttable[bitreverse(word)]==0){

c->dec_firsttablen is clamped to the range 5..8 earlier in the function, so the shift count can be 24..27. However, i is a signed int. When i == 16 and c->dec_firsttablen == 5, the expression shifts 16 left by 27, producing a value that is not representable in signed int. The result is assigned to ogg_uint32_t, but the undefined behavior has already occurred before that conversion.

The observed project stack is:

vorbis_book_init_decode()
_vds_shared_init()
vorbis_synthesis_init()
_make_decode_ready()
_fetch_and_process_packet()
ov_read_filter()
ov_read()
LLVMFuzzerTestOneInput()

PoC

reproducer.poc.ogg.txt

• Please rename reproducer.poc.ogg.txt to reproducer.poc.ogg

Build and reproduction

Vorbis version tested: v1.3.7
OS: Linux x86_64
Compiler: clang
Harness: contrib/oss-fuzz/decode_fuzzer.cc

One working sanitizer configuration is:

export CC=clang
export CXX=clang++
export CFLAGS="-fsanitize=address,undefined,fuzzer-no-link -fno-sanitize-recover=all -fno-omit-frame-pointer -O2 -g"
export CXXFLAGS="-fsanitize=address,undefined,fuzzer-no-link -fno-sanitize-recover=all -fno-omit-frame-pointer -O2 -g"
export ASAN_OPTIONS="detect_leaks=0:abort_on_error=1:symbolize=1"
export UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1"

Build Vorbis with the fuzz harness and run:

./decode_fuzzer <poc>

Driver source

contrib/oss-fuzz/decode_fuzzer.cc:

#include <stdio.h>
#include <string.h>
#include <cstdint>
#include <vorbis/vorbisfile.h>

struct vorbis_data {
  const uint8_t *current;
  const uint8_t *data;
  size_t size;
};

size_t read_func(void *ptr, size_t size1, size_t size2, void *datasource) {
  vorbis_data* vd = (vorbis_data *)(datasource);
  size_t len = size1 * size2;
  if (vd->current + len > vd->data + vd->size) {
    len = vd->data + vd->size - vd->current;
  }
  memcpy(ptr, vd->current, len);
  vd->current += len;
  return len;
}

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
  ov_callbacks memory_callbacks = {0};
  memory_callbacks.read_func = read_func;
  vorbis_data data_st;
  data_st.size = Size;
  data_st.current = Data;
  data_st.data = Data;
  OggVorbis_File vf;
  int result = ov_open_callbacks(&data_st, &vf, NULL, 0, memory_callbacks);
  if (result < 0) {
    return 0;
  }
  int current_section = 0;
  int eof = 0;
  char buf[4096];
  int read_result;
  while (!eof) {
    read_result = ov_read(&vf, buf, sizeof(buf), 0, 2, 1, &current_section);
    if (read_result != OV_HOLE && read_result <= 0) {
      eof = 1;
    }
  }
  ov_clear(&vf);
  return 0;
}

Sanitizer report

sharedbook.c:425:30: runtime error: left shift of 16 by 27 places cannot be represented in type 'int'
    #0 in vorbis_book_init_decode lib/sharedbook.c:425:30
    #1 in _vds_shared_init lib/block.c:240:12
    #2 in vorbis_synthesis_init lib/block.c:709:6
    #3 in _make_decode_ready lib/vorbisfile.c:608:8
    #4 in _fetch_and_process_packet lib/vorbisfile.c:689:15
    #5 in ov_read_filter lib/vorbisfile.c:1977:15
    #6 in ov_read lib/vorbisfile.c:2099:10
    #7 in LLVMFuzzerTestOneInput contrib/oss-fuzz/decode_fuzzer.cc

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior sharedbook.c:425:30 in
SUMMARY: libFuzzer: deadly signal