Garbage output from Quick Start Guide example with release 0.8.0

[StefanoS90]

Describe the bug

Hello, I am trying to run an LLM on the jetson thor.

I was able to make a successful inference of Qwen3 0.6B using the high level api as described in the guide https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/getting_started/quick-start-guide.html#quick-start-guide.

However, the output to the prompt "What is the capital of the United States?" is garbage.
Do you have any clue of what is possibly happaning?

See the input command and the related terminal output below:

(venv) user@user:~/workspace/TensorRT-Edge-LLM$ python - <<'PY'
from experimental.server import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-0.6B")
outputs = llm.generate(
    ["What is the capital of the United States?"],
    SamplingParams(max_tokens=128),
)
print(outputs[0].text)
PY
Downloading (incomplete total...): 0.00B [00:00, ?B/s]                                    Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Downloading (incomplete total...):   2%|▎             | 32.6M/1.52G [00:19<05:06, 4.86MB/s]
Fetching 10 files:  10%|███▌                                | 1/10 [00:00<00:01,  6.89it/s]
Fetching 10 files: 100%|███████████████████████████████████| 10/10 [01:17<00:00,  7.80s/it]
Download complete: 100%|██████████████████████████████| 1.52G/1.52G [01:18<00:00, 19.4MB/s]
[torch.onnx] Obtain model graph for `_Wrapper([...]` with `torch.export.export(..., strict=False)`...
/usr/lib/python3.12/contextlib.py:144: UserWarning: The tensor attribute self._model.model.last_pre_norm_hidden_states was assigned during export. Such attributes must be registered as buffers using the `register_buffer` API (https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_buffer).
  next(self.gen)
[torch.onnx] Obtain model graph for `_Wrapper([...]` with `torch.export.export(..., strict=False)`... ✅
[torch.onnx] Run decompositions...
/usr/lib/python3.12/copyreg.py:99: FutureWarning: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.
  return cls.__new__(cls, *args)
[torch.onnx] Run decompositions... ✅
[torch.onnx] Translate the graph into ONNX...
[torch.onnx] Translate the graph into ONNX... ✅
[torch.onnx] Optimize the ONNX graph...
[torch.onnx] Optimize the ONNX graph... ✅
/home/adas/workspace/TensorRT-Edge-LLM/venv/lib/python3.12/site-packages/torch/onnx/_internal/exporter/_onnx_program.py:486: UserWarning: # The axis name: batch will not be used, since it shares the same shape constraints with another axis: batch.
  rename_mapping = _dynamic_shapes.create_rename_mapping(
/home/adas/workspace/TensorRT-Edge-LLM/venv/lib/python3.12/site-packages/torch/onnx/_internal/exporter/_onnx_program.py:486: UserWarning: # The axis name: past_len will not be used, since it shares the same shape constraints with another axis: past_len.
  rename_mapping = _dynamic_shapes.create_rename_mapping(
[13:47:37.747] [INFO] [llmBuilder.cpp:98:build] Using __LUNOWUD=-peep:match_dual_gemm=off
[13:47:37.747] [INFO] [trtUtils.h:67:loadEdgellmPluginLib] EDGELLM_PLUGIN_PATH variable is not set. Default to build/libNvInfer_edgellm_plugin.so
[13:47:37.865] [INFO] [TensorRT] [MemUsageChange] Init CUDA: CPU -17, GPU +0, now: CPU 1291, GPU 59946 (MiB)
[13:47:39.094] [INFO] [TensorRT] [MemUsageChange] Init builder kernel library: CPU +1227, GPU +1224, now: CPU 2640, GPU 61328 (MiB)
[13:47:39.094] [INFO] [llmBuilder.cpp:128:build] Parsing ONNX model: /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/model.onnx
[13:47:39.100] [INFO] [TensorRT] ----------------------------------------------------------------
[13:47:39.100] [INFO] [TensorRT] Input filename:   /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/model.onnx
[13:47:39.101] [INFO] [TensorRT] ONNX IR version:  0.0.10
[13:47:39.101] [INFO] [TensorRT] Opset version:    24
[13:47:39.101] [INFO] [TensorRT] Producer name:    pytorch
[13:47:39.101] [INFO] [TensorRT] Producer version: 2.12.0+cu130
[13:47:39.101] [INFO] [TensorRT] Domain:           
[13:47:39.101] [INFO] [TensorRT] Model version:    0
[13:47:39.101] [INFO] [TensorRT] Doc string:       
[13:47:39.101] [INFO] [TensorRT] ----------------------------------------------------------------
[13:47:39.101] [WARNING] [TensorRT] ModelImporter.cpp:653: Make sure input last_token_ids has Int64 binding.
[13:47:39.104] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.120] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.121] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.121] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.122] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.122] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.124] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.124] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.125] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.125] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.126] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.126] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.127] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.127] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.128] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.128] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.129] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.129] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.130] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.130] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.131] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.131] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.133] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.133] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.134] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.134] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.135] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.135] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.136] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.136] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.137] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.137] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.138] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.138] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.140] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.140] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.141] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.141] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.142] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.142] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.143] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.143] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.144] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.144] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.145] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.145] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.146] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.146] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.147] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.148] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.149] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.149] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.150] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.150] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.152] [INFO] [TensorRT] Searching for plugin: AttentionPlugin, plugin_version: 1, plugin_namespace: 
[13:47:39.152] [INFO] [TensorRT] Successfully created plugin: AttentionPlugin
[13:47:39.268] [INFO] [TensorRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[13:47:39.357] [INFO] [TensorRT] Compiler backend is used during engine build.
[13:48:10.366] [INFO] [TensorRT] Detected 33 inputs and 29 output network tensors.
[13:48:10.689] [INFO] [TensorRT] Total Host Persistent Memory: 80 bytes
[13:48:10.689] [INFO] [TensorRT] Total Device Persistent Memory: 0 bytes
[13:48:10.689] [INFO] [TensorRT] Max Scratch Memory: 117441024 bytes
[13:48:10.689] [INFO] [TensorRT] [BlockAssignment] Started assigning block shifts. This will take 1 steps to complete.
[13:48:10.689] [INFO] [TensorRT] [BlockAssignment] Algorithm ShiftNTopDown took 0.01038ms to assign 1 blocks to 1 nodes requiring 117441024 bytes.
[13:48:10.689] [INFO] [TensorRT] Total Activation Memory: 117441024 bytes
[13:48:26.713] [INFO] [TensorRT] Detected 33 inputs and 29 output network tensors.
[13:48:26.969] [INFO] [TensorRT] Total Host Persistent Memory: 80 bytes
[13:48:26.969] [INFO] [TensorRT] Total Device Persistent Memory: 0 bytes
[13:48:26.969] [INFO] [TensorRT] Max Scratch Memory: 33575424 bytes
[13:48:26.969] [INFO] [TensorRT] [BlockAssignment] Started assigning block shifts. This will take 1 steps to complete.
[13:48:26.969] [INFO] [TensorRT] [BlockAssignment] Algorithm ShiftNTopDown took 0.00837ms to assign 1 blocks to 1 nodes requiring 33575424 bytes.
[13:48:26.969] [INFO] [TensorRT] Total Activation Memory: 33575424 bytes
[13:48:27.043] [INFO] [TensorRT] Total Weights Memory: 1192100096 bytes
[13:48:27.049] [INFO] [TensorRT] Compiler backend is used during engine execution.
[13:48:27.049] [INFO] [TensorRT] Engine generation completed in 47.7824 seconds.
[13:48:27.050] [INFO] [TensorRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1136 MiB
[13:48:27.210] [INFO] [TensorRT] [MemUsageStats] Peak memory usage during Engine building and serialization: CPU: 13828 MiB
[13:48:27.826] [INFO] [builderUtils.cpp:328:buildAndSerializeEngine] Engine saved to /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/llm.engine
[13:48:27.829] [INFO] [llmBuilder.cpp:944:copyConfig] Copied config.json with builder config to /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/config.json
[13:48:27.829] [INFO] [fileUtils.cpp:48:copyFile] Successfully copied /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/tokenizer_config.json to /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/tokenizer_config.json
[13:48:27.829] [INFO] [llmBuilder.cpp:975:copyTokenizerFiles] Copied tokenizer file: tokenizer_config.json
[13:48:27.834] [INFO] [fileUtils.cpp:48:copyFile] Successfully copied /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/tokenizer.json to /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/tokenizer.json
[13:48:27.834] [INFO] [llmBuilder.cpp:975:copyTokenizerFiles] Copied tokenizer file: tokenizer.json
[13:48:27.834] [INFO] [fileUtils.cpp:48:copyFile] Successfully copied /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/processed_chat_template.json to /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/processed_chat_template.json
[13:48:27.834] [INFO] [llmBuilder.cpp:975:copyTokenizerFiles] Copied tokenizer file: processed_chat_template.json
[13:48:27.973] [INFO] [fileUtils.cpp:48:copyFile] Successfully copied /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/embedding.safetensors to /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/embedding.safetensors
[13:48:27.973] [INFO] [llmBuilder.cpp:1129:copyEmbeddingFile] Copied embedding.safetensors to /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/embedding.safetensors
[13:48:29.281] [INFO] [trtUtils.h:67:loadEdgellmPluginLib] EDGELLM_PLUGIN_PATH variable is not set. Default to build/libNvInfer_edgellm_plugin.so
[13:48:29.402] [INFO] [llmRuntimeUtils.cpp:444:loadEmbeddingTable] Loaded FP16 embedding: [151936, 1024]
[13:48:29.402] [INFO] [llmEngineConfig.cpp:229:parseEngineConfig] reading /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/config.json
[13:48:29.404] [INFO] [llmRuntimeUtils.cpp:167:collectRopeConfig] Collected rope config: RopeConfig:  type: Default  rotaryScale: 1  rotaryTheta: 1e+06  maxPositionEmbeddings: 40960
[13:48:29.404] [INFO] [llmEngineConfig.cpp:331:parseEngineConfig] LLMEngineConfig{ hiddenSize=1024 vocabSize=151936 outputVocabSize=151936 numDecoderLayers=28 numAttentionLayers=28 numKVHeads=8 headDim=128 rotaryDim=128 maxBatch=1 maxInputLen=4096 maxKVCapacity=8192 useTrtNativeOps=false isSpecDecodeBase=false specDecodeType=0 loraRank=0 }
[13:48:29.407] [INFO] [engineExecutor.cpp:36:EngineExecutor] loading engine file: /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/llm.engine
[13:48:29.407] [INFO] [TensorRT] Loaded engine size: 1140 MiB
[13:48:29.538] [INFO] [TensorRT] [MS] Running engine with multi stream info
[13:48:29.538] [INFO] [TensorRT] [MS] Number of aux streams is 1
[13:48:29.538] [INFO] [TensorRT] [MS] Number of total worker streams is 2
[13:48:29.538] [INFO] [TensorRT] [MS] The main stream provided by execute/enqueue calls is the first worker stream
[13:48:29.626] [INFO] [TensorRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 1136 (MiB)
[13:48:29.626] [INFO] [engineExecutor.cpp:53:EngineExecutor] engine loaded successfully (62 I/O tensors)
[13:48:29.731] [INFO] [llmInferenceRuntime.cpp:118:initializeCommon] Base EngineExecutor successfully loaded from /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/llm.engine.
[13:48:29.731] [INFO] [llmInferenceRuntime.cpp:129:initializeCommon] Runtime batch size set to: 1 (from engine bundle)
[13:48:29.742] [INFO] [ropeCache.cpp:103:getOrCreate] RopeCache: creating new entry (rotaryDim=128, maxSeqLen=8192)
[13:48:29.753] [INFO] [llmInferenceRuntime.cpp:244:initializeCommon] Runtime tensors successfully allocated.
[13:48:29.753] [INFO] [llmInferenceRuntime.cpp:272:initializeCommon] Start loading tokenizer from model directory: /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm
[13:48:31.837] [INFO] [tokenizer.cpp:385:loadVocabulary] Loaded 151643 vocabulary tokens
[13:48:32.044] [INFO] [tokenizer.cpp:96:loadFromHF] Loaded 26 special tokens
[13:48:32.163] [INFO] [tokenizer.cpp:782:loadChatTemplate] Successfully loaded chat template from /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm/processed_chat_template.json (for model: /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca)
[13:48:32.163] [INFO] [tokenizer.cpp:123:loadFromHF] Successfully loaded tokenizer from /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm (vocab_size=151669)
[13:48:32.191] [INFO] [llmInferenceRuntime.cpp:274:initializeCommon] Tokenizer successfully loaded from model directory: /home/adas/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca/.edgellm/onnx/llm/.edgellm/engine/i4096_b1_kv8192/llm
[13:48:32.193] [INFO] [llmInferenceRuntime.cpp:372:initializeCommon] Setup shared execution context memory: 117441024 bytes (base requires: 117441024, strategy requires: 0, vision requires: 0, audio requires: 0, action requires: 0)
[13:48:32.194] [INFO] [TensorRT] Switching optimization profile from: 0 to 1. Please ensure there are no enqueued operations pending in this context prior to switching profiles
[13:48:32.294] [INFO] [engineExecutor.cpp:219:captureGraph] captured graph (hash=0xd5d500c02b15a11f)
[13:48:32.294] [INFO] [decoderRegistry.cpp:84:captureCudaGraphs] Successfully captured decoding CUDA graphs for active decoding strategies.
[13:48:32.295] [INFO] [TensorRT] Switching optimization profile from: 1 to 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
[13:48:32.331] [INFO] [TensorRT] Switching optimization profile from: 0 to 1. Please ensure there are no enqueued operations pending in this context prior to switching profiles
lec





1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1

Steps/Code to reproduce bug

Build configuration:

mkdir -p build
cd build
cmake .. \
  -DTRT_PACKAGE_DIR=/usr \
  -DCUDA_CTK_VERSION=13.0 \
  -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
  -DEMBEDDED_TARGET=jetson-thor \
  -DENABLE_CUTE_DSL=ALL \
  -DBUILD_PYTHON_BINDINGS=ON
make -j$(nproc)
cd ..

Runtime command used:

python - <<'PY'
from experimental.server import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-0.6B")
outputs = llm.generate(
    ["What is the capital of the United States?"],
    SamplingParams(max_tokens=128),
)
print(outputs[0].text)
PY

Expected behavior

i would expect correct text answer, but the actual output is

lec

1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1

System information Edge Device

• Platform: NVIDIA Jetson AGX Thor Developer Kit
• JetPack package version: 7.1-b112
• L4T / Jetson Linux release: # R38 (release), REVISION: 4.0, GCID: 43443517, BOARD: generic, EABI: aarch64, DATE: Wed Dec 31 00:15:19 UTC 2025
  KERNEL_VARIANT: oot
  TARGET_USERSPACE_LIB_DIR=nvidia
  TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
  INSTALL_TYPE=
• CPU architecture: aarch64
• GPU compute capability: SM110
• Total device memory: 122Gi
• Build type: Release
• Library versions:
  • TensorRT Edge-LLM version or commit hash: ?
  • CUDA: 13.0
  • TensorRT: 10.13.3
  • C++ compiler: GCC 13.3.0
• CMake options used:
  • CMAKE_TOOLCHAIN_FILE: ?
  • EMBEDDED_TARGET: ?
  • TRT_PACKAGE_DIR: ?
• Any other details that may help: ?
  ======================================================================

[StefanoS90]

Just one more info: I just tested with the release 0.7.1 seem to be working correctly.

[nvluxiaoz]

Thanks. We will investigate this issue.

[yangkaiHandsome]

I'm experiencing the same issue on my Jetson Thor device with the same environment:

• JetPack: 7.1 (R38.2)
• CUDA: 13.0
• TensorRT: 10.13.3.9
• TensorRT-Edge-LLM: 0.8.0

In my case, I'm running the Qwen3-ASR-0.6B model following the ASR example guide. The export, file transfer, engine build, and audio preprocessing all complete successfully without errors. However, the inference output is garbled/repetitive nonsense, e.g.:

"iding.....�.\n\n\n\n.....le\n\n\n.com.ating.ating.ating.ating..."

I verified the following:

• The ONNX files are identical on x86 and Thor (MD5 checksums match)
• The same model produces correct output when running directly with HuggingFace Transformers on x86 GPU (e.g., "这是一段测试语音")
• Tried multiple audio files (3s and 11s) with the same garbled result
• Tried both temperature=1.0 and temperature=0.0 â�� no difference

This confirms the issue is in the TensorRT engine inference on JetPack 7.0/7.1, not in the model or export pipeline.

Is there a known workaround, or will this be addressed in a future release? Would upgrading to JetPack 7.2 resolve the issue?

[yangkaiHandsome]

Same root cause on Jetson Thor (sm_110), reproduced with Qwen3-TTS � not model-specific

I'm hitting what looks like the exact same bug as this issue, but with Qwen3-TTS-12Hz-1.7B-CustomVoice (and earlier with Qwen3-ASR). On Jetson Thor the model runs end to end without crashing, but the output is garbage: the Talker never emits EOS and keeps generating degenerate/repeating codec tokens until it hits max_audio_length, producing ~5 minutes of noise for a sentence that should be ~5 seconds. This matches the "lec" / "1.1.1..." degeneration reported here for plain Qwen3-0.6B, so it does not appear to be model-, export-, or precision-specific � it affects every model on this device.

Environment

• Device: NVIDIA Jetson AGX Thor (compute capability sm_110)
• JetPack 7.1, CUDA 13.0
• TensorRT Edge-LLM v0.8.0 (latest main, f9cc746)
• CMake config: -DEMBEDDED_TARGET=jetson-thor -DCUDA_CTK_VERSION=13.0 -DENABLE_CUTE_DSL=ALL -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake, CUTE_DSL_ARTIFACT_TAG left to auto (only cuteDSLArtifact/aarch64/sm_110 is present, so the correct tag is selected).

Things I ruled out (so this is almost certainly a runtime/kernel issue on sm_110, not user setup)

• ONNX export structure and all three config.json files are correct (model_type: qwen3_tts_talker, EOS/speaker IDs present).
• Checkpoint weights load fully; the "skipped" tensors are exactly the nested CodePredictor weights handled by the separate export pass.
• No FP16 weight overflow â�� max abs Talker weight is ~153, well within FP16 range.
• code_predictor / code2wav engines are correctly auto-located and used.
• Build configuration and CuTe DSL artifact tag are correct for Thor.

What looks like the root cause

The prebuilt context FMHA cubins do not include an sm_110 variant � only:
sm80, sm86, sm87, sm89, sm100, sm101, sm120, sm121

And the kernel-selection guard in cpp/kernels/contextAttentionKernels/contextFMHARunner.cpp does not account for sm_110:
bool const isSm8x = (smVersion == kSM_80 || kSM_86 || kSM_87 || kSM_89);
bool const isSm10x = (smVersion == kSM_100 || kSM_101);
bool const isSm12x = (smVersion == kSM_120 || kSM_121);
check::check((isSm8x || isSm10x || isSm12x),
"Other SMs are not supported by context FMHA-v2 kernels");
sm_110 (Thor) falls into none of these buckets. CuteDslFMHARunner::canImplement does claim support for smVersion >= 100, and an sm_110 CuTe DSL artifact ships in the repo, but the end-to-end output on Thor is still garbage � so either the sm_110 path isn't actually exercised for attention, or the sm_110 kernels produce incorrect results.

Questions for the maintainers

1. Is Jetson Thor (sm_110) intended to be supported in v0.8.0? The shipping cuteDSLArtifact/aarch64/sm_110 suggests yes, but the context-FMHA cubins and SM guards exclude it.
2. Is there a supported workaround to get correct attention numerics on sm_110 today (e.g., forcing a specific FMHA backend)?
3. If this is a known gap, is a fix planned, and is there a recommended interim path (e.g., run on Orin / x86 with the same exported ONNX)?

Happy to provide full logs, the exported configs, or run any diagnostic build you suggest.

[yangkaiHandsome]

Update / workaround:

After further testing, I found that this issue does not occur with TensorRT Edge-LLM v0.7.1 on the same Jetson Thor environment.

With TensorRT Edge-LLM v0.7.1, Qwen3-ASR-0.6B produces correct inference results. The garbled / repetitive output only appears after switching to TensorRT Edge-LLM v0.8.0.

So for anyone running into the same problem on Jetson Thor, please be careful with v0.8.0. This may be a regression in v0.8.0 related to the Thor / sm_110 runtime or attention kernel path.

Current workaround:

Use TensorRT Edge-LLM v0.7.1 instead of v0.8.0.

I’m leaving this note here to help others avoid spending time debugging the model export, ONNX files, audio preprocessing, or checkpoints. In my case, those parts were not the root cause.