Describe the bug
When I run a qemu-tdx(ubuntu qemu-8.2.x) within kata and hotplug block device via QMP with virtio-blk-pci, command as below:
/usr/bin/qemu-system-x86_64 -name sandbox-a7c8d5202864919353edce71cd77430818f165d1d2b8154a9b585b5676618759 -kernel /opt/kata/share/kata-containers/vmlinuz-6.12.13-150-confidential -append reboot=k panic=1 systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service root=/dev/vda1 rootflags=data=ordered,errors=remount-ro ro rootfstype=ext4 agent.debug_console agent.debug_console_vport=1026 cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 selinux=0 -smp 1,maxcpus=128 -machine q35,accel=kvm,confidential-guest-support=tdx -cpu host,pmu=off -m 2G,slots=128,maxmem=128340M -qmp unix:fd=14,server=on,wait=off -rtc base=utc,clock=host,driftfix=slew -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=off,addr=2,io-reserve=4k,mem-reserve=1m,pref64-reserve=1m -device pci-bridge,bus=pcie.0,id=pci-bridge-1,chassis_nr=2,shpc=off,addr=3,io-reserve=4k,mem-reserve=1m,pref64-reserve=1m -device vhost-vsock-pci,vhostfd=15,guest-cid=572951779 -netdev tap,id=network-tap0_kata,vhost=on,vhostfds=18,fds=17 -device virtio-net-pci,netdev=network-tap0_kata,mac=ae:6b:29:3c:14:b4,mq=on,vectors=4 -blockdev driver=file,node-name=image-d98e9eabafe197b8,filename=/opt/kata/share/kata-containers/kata-ubuntu-oracular-confidential.image,aio=threads,cache.direct=off,cache.no-flush=off,auto-read-only=on -device virtio-blk-pci,drive=image-d98e9eabafe197b8,config-wce=off,share-rw=on,serial=image-d98e9eabafe197b8 -object {"qom-type":"tdx-guest","id":"tdx","sept-ve-disable":true,"quote-generation-socket":{"type":"vsock","cid":"2","port":"4050"}} -bios /usr/share/ovmf/OVMF.fd -device virtio-serial-pci,id=serial0 -device virtconsole,id=console0,chardev=charconsole0 -chardev socket,id=charconsole0,server=on,wait=off,path=console.sock -vga none -no-user-config -nodefaults -nographic -no-reboot
The block device hot plugged and return its pci slot, but there's no real /dev/vdX found in guest. And I do dmesg, some exceptional info comes up:
pci0000:00/0000:00:02.0/0000:01:00.0# dmesg |grep 0000:01:00.0
[ 8.079069] pci 0000:01:00.0: [1af4:1042] type 00 class 0x010000 conventional PCI endpoint
[ 8.080218] pci 0000:01:00.0: BAR 1 [mem 0x00000000-0x00000fff]
[ 8.081410] pci 0000:01:00.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]
[ 8.085773] pci 0000:01:00.0: BAR 4 [mem size 0x00004000 64bit pref]: can't assign; no space
[ 8.085775] pci 0000:01:00.0: BAR 4 [mem size 0x00004000 64bit pref]: failed to assign
[ 8.085777] pci 0000:01:00.0: BAR 1 [mem size 0x00001000]: can't assign; no space
[ 8.085778] pci 0000:01:00.0: BAR 1 [mem size 0x00001000]: failed to assign
[ 8.088246] virtio-pci 0000:01:00.0: virtio_pci: leaving for legacy driver
And I have tested on non-TEE ENV, it works well:
root@localhost:/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0# dmesg |grep 0000:01:00.0
[ 0.489003] pci 0000:01:00.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint
[ 0.489103] pci 0000:01:00.0: BAR 0 [io 0x0000-0x007f]
[ 0.489138] pci 0000:01:00.0: BAR 1 [mem 0x00000000-0x00000fff]
[ 0.489260] pci 0000:01:00.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]
[ 0.489996] pci 0000:01:00.0: BAR 4 [mem 0xfe800000-0xfe803fff 64bit pref]: assigned
[ 0.490046] pci 0000:01:00.0: BAR 1 [mem 0xfe200000-0xfe200fff]: assigned
[ 0.490063] pci 0000:01:00.0: BAR 0 [io 0xd000-0xd07f]: assigned
[ 0.490696] virtio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[ 0.495329] probe of 0000:01:00.0 returned 0 after 5243 usecs
How to address this problem, please for your help, Thx
System report
root@intel-tdx-ArcherCity:/home/intel-tdx/tdx# ./system-report.sh
If you are running this for reporting an issue on GitHub,
copy all output between the markers below.
<======== COPY BELOW HERE ========>
### Git ref
d35945b
### Operating system details
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
6.8.0-1015-intel #22-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 12 14:47:29 UTC 2024 x86_64 x86_64 GNU/Linux
...
### TDX CPU instruction support
CPU supports TDX according to /proc/cpuinfo
### Model specific registers (MSRs)
MK_TME_ENABLED bit: 1 (expected value: 1)
SEAM_RR bit: 1 (expected value: 1)
NUM_TDX_PRIV_KEYS: 20
SGX_AND_MCHECK_STATUS: 0 (expected value: 0)
Production platform: Pre-production (expected value: Production)
Genuine Intel(R) 0000
Status: Installed
Package: qemu-system-x86
Version: 2:8.2.2+ds-0ubuntu1.4+tdx1.1
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages
### Libvirt package details
Status: Installed
Package: libvirt-clients
Version: 10.0.0-2ubuntu8.3+tdx1.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages
Status: Installed
Package: ovmf
Version: 2024.02-3+tdx1.0
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages
### sgx-dcap-pccs package details
Status: Installed
Package: sgx-dcap-pccs
Version: 1.21-0ubuntu1
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages
### tdx-qgs package details
Status: Installed
Package: tdx-qgs
Version: 1.21-0ubuntu2.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages
### sgx-ra-service package details
Status: Installed
Package: sgx-ra-service
Version: 1.21-0ubuntu2.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages
Description: Intel(R) Software Guard Extensions Multi-Package Registration Agent Service
### sgx-pck-id-retrieval-tool package details
Status: Installed
Package: sgx-pck-id-retrieval-tool
Version: 1.21-0ubuntu2.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages
● qgsd.service - Intel(R) TD Quoting Generation Service
Loaded: loaded (/usr/lib/systemd/system/qgsd.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-04-15 17:52:26 CST; 1 month 5 days ago
Main PID: 2393305 (qgs)
Tasks: 5 (limit: 153917)
Memory: 920.0K (peak: 4.7M)
CPU: 113ms
CGroup: /system.slice/qgsd.service
└─2393305 /usr/bin/qgs
Notice: journal has been rotated since unit was started, output may be incomplete.
● pccs.service - Provisioning Certificate Caching Service (PCCS)
Loaded: loaded (/usr/lib/systemd/system/pccs.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-04-15 17:52:27 CST; 1 month 5 days ago
Docs: https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/master/QuoteGeneration/pccs/README.md
Main PID: 2393559 (node)
Tasks: 15 (limit: 153917)
Memory: 71.9M (peak: 74.3M)
CPU: 51.840s
CGroup: /system.slice/pccs.service
└─2393559 /usr/bin/node /opt/intel/sgx-dcap-pccs/pccs_server.js
May 21 01:00:02 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:02.741 [info]: Request-ID is : 05c93df74ba446c684fdae36a973cb32
May 21 01:00:03 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:03.928 [info]: Request-ID is : 2fd1770abd534a9696743ffecd7008b6
May 21 01:00:07 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:07.175 [info]: Request-ID is : 2d2e5df0056540fa8b9d97c9981e0a66
May 21 01:00:08 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:08.204 [info]: Request-ID is : f1c1567f403e4424ad007a73b453ccae
May 21 01:00:09 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:09.249 [info]: Request-ID is : 0b910047cba14e98b6ce44ae612a355a
May 21 01:00:10 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:10.340 [info]: Request-ID is : bed9251bab2b4bfeae87e82001d7fd12
May 21 01:00:13 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:13.633 [info]: Request-ID is : 187c600d95ae4f7ca737b78f8248780a
May 21 01:00:15 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:15.111 [info]: Request-ID is : 0d8c0e9720db4c46965aef1bd12a558b
May 21 01:00:16 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:16.210 [info]: Request-ID is : 446f74708ec14b389ed7fb9d7fe27c14
May 21 01:00:16 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:16.876 [info]: Scheduled cache refresh is completed successfully.
### MPA registration logs (last 30 lines)
[15-04-2025 05:52:38] INFO: SGX Registration Agent version: 1.21.100.3
[15-04-2025 05:52:38] INFO: Starts Registration Agent Flow.
[15-04-2025 05:52:38] INFO: Registration Flow - Registration status indicates registration is completed successfully. MPA has nothing to do.
[15-04-2025 05:52:38] INFO: Finished Registration Agent Flow.
<======== COPY ABOVE HERE ========>
Related Issues
Describe the bug
When I run a qemu-tdx(ubuntu qemu-8.2.x) within kata and hotplug block device via QMP with virtio-blk-pci, command as below:
The block device hot plugged and return its pci slot, but there's no real /dev/vdX found in guest. And I do dmesg, some exceptional info comes up:
And I have tested on non-TEE ENV, it works well:
How to address this problem, please for your help, Thx
System report
d35945b
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
6.8.0-1015-intel #22-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 12 14:47:29 UTC 2024 x86_64 x86_64 GNU/Linux
...
CPU supports TDX according to /proc/cpuinfo
MK_TME_ENABLED bit: 1 (expected value: 1)
SEAM_RR bit: 1 (expected value: 1)
NUM_TDX_PRIV_KEYS: 20
SGX_AND_MCHECK_STATUS: 0 (expected value: 0)
Production platform: Pre-production (expected value: Production)
Genuine Intel(R) 0000
Status: Installed
Package: qemu-system-x86
Version: 2:8.2.2+ds-0ubuntu1.4+tdx1.1
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages
Status: Installed
Package: libvirt-clients
Version: 10.0.0-2ubuntu8.3+tdx1.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages
Status: Installed
Package: ovmf
Version: 2024.02-3+tdx1.0
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages
Status: Installed
Package: sgx-dcap-pccs
Version: 1.21-0ubuntu1
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages
Status: Installed
Package: tdx-qgs
Version: 1.21-0ubuntu2.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages
Status: Installed
Package: sgx-ra-service
Version: 1.21-0ubuntu2.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages
Description: Intel(R) Software Guard Extensions Multi-Package Registration Agent Service
Status: Installed
Package: sgx-pck-id-retrieval-tool
Version: 1.21-0ubuntu2.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages
● qgsd.service - Intel(R) TD Quoting Generation Service
Loaded: loaded (/usr/lib/systemd/system/qgsd.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-04-15 17:52:26 CST; 1 month 5 days ago
Main PID: 2393305 (qgs)
Tasks: 5 (limit: 153917)
Memory: 920.0K (peak: 4.7M)
CPU: 113ms
CGroup: /system.slice/qgsd.service
└─2393305 /usr/bin/qgs
Notice: journal has been rotated since unit was started, output may be incomplete.
● pccs.service - Provisioning Certificate Caching Service (PCCS)
Loaded: loaded (/usr/lib/systemd/system/pccs.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-04-15 17:52:27 CST; 1 month 5 days ago
Docs: https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/master/QuoteGeneration/pccs/README.md
Main PID: 2393559 (node)
Tasks: 15 (limit: 153917)
Memory: 71.9M (peak: 74.3M)
CPU: 51.840s
CGroup: /system.slice/pccs.service
└─2393559 /usr/bin/node /opt/intel/sgx-dcap-pccs/pccs_server.js
May 21 01:00:02 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:02.741 [info]: Request-ID is : 05c93df74ba446c684fdae36a973cb32
May 21 01:00:03 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:03.928 [info]: Request-ID is : 2fd1770abd534a9696743ffecd7008b6
May 21 01:00:07 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:07.175 [info]: Request-ID is : 2d2e5df0056540fa8b9d97c9981e0a66
May 21 01:00:08 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:08.204 [info]: Request-ID is : f1c1567f403e4424ad007a73b453ccae
May 21 01:00:09 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:09.249 [info]: Request-ID is : 0b910047cba14e98b6ce44ae612a355a
May 21 01:00:10 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:10.340 [info]: Request-ID is : bed9251bab2b4bfeae87e82001d7fd12
May 21 01:00:13 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:13.633 [info]: Request-ID is : 187c600d95ae4f7ca737b78f8248780a
May 21 01:00:15 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:15.111 [info]: Request-ID is : 0d8c0e9720db4c46965aef1bd12a558b
May 21 01:00:16 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:16.210 [info]: Request-ID is : 446f74708ec14b389ed7fb9d7fe27c14
May 21 01:00:16 intel-tdx-ArcherCity node[2393559]: 2025-05-21 01:00:16.876 [info]: Scheduled cache refresh is completed successfully.
[15-04-2025 05:52:38] INFO: SGX Registration Agent version: 1.21.100.3
[15-04-2025 05:52:38] INFO: Starts Registration Agent Flow.
[15-04-2025 05:52:38] INFO: Registration Flow - Registration status indicates registration is completed successfully. MPA has nothing to do.
[15-04-2025 05:52:38] INFO: Finished Registration Agent Flow.
Related Issues
block_device_driver = "virtio-blk"problems kata-containers/kata-containers#11238