diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index 2931d4a36..63091f890 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -28,6 +28,18 @@ jobs: run: sudo apt-get install -y lld - name: Build and test (generic) run: ./ci/jobs/build-and-test-${{ matrix.profile }}.sh + build-and-test-linux-aarch64: + # The AArch64 recompiler runs through the generic sandbox here, at a native 4K page size -- + # which macOS (16K) cannot exercise. The Linux zygote sandbox is x86_64-only, hence generic. + runs-on: ubuntu-24.04-arm + steps: + - uses: actions/checkout@v4 + - name: uname -a + run: uname -a + - name: Install LLD + run: sudo apt-get install -y lld + - name: Build and test (generic sandbox, AArch64) + run: ./ci/jobs/build-and-test-linux-aarch64.sh build-and-test-linux-only: runs-on: ubuntu-24.04 steps: diff --git a/RECOMPILER_ARCHITECTURE.md b/RECOMPILER_ARCHITECTURE.md new file mode 100644 index 000000000..32cba037b --- /dev/null +++ b/RECOMPILER_ARCHITECTURE.md @@ -0,0 +1,176 @@ +# Recompiler architecture + +The "recompiler" (a.k.a. compiler) translates PolkaVM guest bytecode into native machine code at module-load time. It targets x86-64 today, and runs the compiled code inside one of two sandboxes (Linux or generic). Below is an end-to-end walkthrough of how that pipeline is wired together. + +## 1. Big picture + +``` +ProgramBlob ──► Module::from_blob ──► compile_module! ──► CompilerVisitor (ParsingVisitor) + │ + │ per-instruction emit via ArchVisitor (amd64.rs) + ▼ + Assembler buffer + │ finish_compilation: fixups, gas costs, trampolines + ▼ + CompiledModule + │ spawn_and_load_module + ▼ + Sandbox instance (Linux | generic) + │ run() → futex/jump into native code → InterruptKind + ▼ + Caller (host) +``` + +Three layers cooperate: + +1. **Front-end driver** (`crates/polkavm/src/api.rs`, `compiler.rs`) — owns the compilation lifecycle and the instruction-visitor loop. +2. **Back-end codegen** (`compiler/amd64.rs` + `polkavm-assembler`) — knows about x86-64 instruction encoding, the register map, and the calling-convention boilerplate. +3. **Sandbox runtime** (`sandbox.rs`, `sandbox/{linux,generic}.rs`, `polkavm-zygote`) — owns the address space the compiled code runs in, page-fault recovery, and the host↔guest control transfer. + +## 2. The driver loop + +Entry points are `Module::new` / `Module::from_blob` in `crates/polkavm/src/api.rs:397,403`. After validating the blob and consulting the module cache, the macro `compile_module!` at `api.rs:525` picks the concrete generic instantiation — `(Sandbox = Linux | Generic) × (Bitness = 32 | 64) × (GasVisitor = simple | Simulator)` — and: + +1. Instantiates `CompilerVisitor<'a, S, B, G>` (`compiler.rs:71`), passing the assembler, gas visitor, and program metadata. +2. Calls `blob.visit(&mut visitor)`, which iterates every instruction in the blob and dispatches to the visitor's `ParsingVisitor` impl. Each guest opcode (e.g. `add_32`, `load_u32`, `branch_eq`, `ecalli`) becomes one method call. +3. Finalizes with `CompilerVisitor::finish_compilation` (`compiler.rs:308`) — patches forward jumps, lays out the trampolines, records per-block gas costs and the PC↔native-offset map, and emits the final `CompiledModule`. + +Important state held on `CompilerVisitor`: + +| Field | Purpose | +| --- | --- | +| `asm: Assembler` | x86-64 instruction buffer (`polkavm-assembler`) | +| `program_counter_to_label: FlatMap` | guest PC → assembler `Label` | +| `gas_visitor: G` | per-instruction cost accumulator | +| pre-allocated labels (`ecall_label`, `trap_label`, `sbrk_label`, `memset_label`, `divrem_label`, `step_label`) | targets for shared trampolines emitted once at the end | + +The fact that every handler is dispatched through a single `ParsingVisitor` impl is what lets the compiler stay relatively small (~2 kloc of x86 codegen for the whole ISA): nothing decodes bytecode separately — the upstream `polkavm-common` parser hands typed operands directly to each handler. + +## 3. Per-instruction code generation + +The actual machine-code emission lives in `crates/polkavm/src/compiler/amd64.rs`. It implements `ArchVisitor` for `CompilerVisitor` and adds an x86-64-specific layer on top of `polkavm-assembler`. + +**Register map.** `REG_MAP[16]` at `amd64.rs:52-69` pins each of the 13 guest registers onto a native register; the conversion is `conv_reg(RawReg) → polkavm_common::regmap::to_native_reg`. Two native registers are reserved as scratch (`TMP_REG`, `AUX_TMP_REG`); on the generic sandbox `AUX_TMP_REG` doubles as the guest-memory base pointer (`GENERIC_SANDBOX_MEMORY_REG`). + +**Operand addressing.** A macro `load_store_operand!` at `amd64.rs:95-148` expands every load/store into either: +- a direct `[base + disp]` form on Linux (because guest memory sits at a known offset inside the same address space the compiled code lives in), or +- a `[memory_reg + disp]` form on the generic sandbox (where guest memory is reached through an explicit base pointer at `vmctx_ptr - 4096`, see `generic.rs:120`). + +This is the only divergence in codegen between the two sandboxes — most handlers are sandbox-agnostic. + +**Representative handlers.** + +- **Arithmetic.** `add_32` (`amd64.rs:1558`) → `add_generic` (`amd64.rs:1530`): reserve two assembler slots, convert operands, emit `add(RegSize::R32, d, s1, s2)`. Multi-operand arithmetic always reuses the same three-step pattern: reserve, convert, emit. +- **Memory.** `load` (`amd64.rs:345`) expands `load_store_operand!`, then emits a single `mov` of the right width (U8/I8/…/U64). Sign-/zero-extension is part of the load kind itself. +- **Control flow.** `branch` (`amd64.rs:483`) converts both operands, looks up (or lazily creates) the label for the target PC in `program_counter_to_label`, and calls `branch_to_label` (`amd64.rs:180`), which emits a conditional jump and records a `Fixup` for the assembler to resolve at the end. +- **Hostcall.** `ecalli` (`amd64.rs:1354`) stores the current guest PC, the call number and the next PC into the VM context, then `call`s the shared `ecall_label` trampoline. The trampoline (`emit_ecall_trampoline`, `amd64.rs:625`) saves the registers into the VM context, flips the futex to `VMCTX_FUTEX_GUEST_ECALLI`, and parks on it until the host either resumes execution or unwinds via `VMCTX_FUTEX_LONGJUMP`. + +**Basic blocks and gas metering.** Block boundaries are managed by `force_start_new_basic_block` (`compiler.rs:470`) and `after_instruction` (`compiler.rs:500-539`). At each block start, when gas metering is on, `emit_gas_metering_stub` (`amd64.rs:942-978`) emits: + +``` +sub qword [vmctx + gas_offset], i32::MAX ; the literal is patched to the real block cost +``` + +In synchronous metering mode, the stub additionally encodes a short backward branch into an illegal opcode so that gas-exhaustion turns into a SIGILL the sandbox can catch. The real per-block cost is written at the end of compilation, when `gas_visitor.take_block_cost()` is called. + +**Invalid jumps.** Jump-table entries that don't correspond to a real target are stamped with `JUMP_TABLE_INVALID_ADDRESS = 0xfa6f29540376ba8a` (`compiler.rs:42`) — an address that's guaranteed not to map. Jumping there segfaults, and the sandbox turns the segfault into `InterruptKind::Trap` rather than killing the process. + +## 4. The assembler (`polkavm-assembler`) + +The recompiler does not synthesise bytes itself; it drives `polkavm-assembler::Assembler` (`crates/polkavm-assembler/src/assembler.rs:12`). `Assembler` holds: + +- `code: Vec` — the emitted bytes, +- `labels: Vec` — resolved label offsets, +- `fixups: Vec` — forward references waiting to be patched. + +A notable design point is the **reserved-slot pattern**: `asm.reserve::()` returns a `ReservedAssembler` typed by the number of slots remaining. Each `push(inst)` consumes one slot statically (returns `U1`, then `U0`); `assert_reserved_exactly_as_needed()` ensures every handler pushes exactly what it reserved. That removes a class of "forgot to reserve enough room" bugs from the codegen and lets the assembler keep a single contiguous buffer (no resizing surprises in the middle of an instruction). + +Labels declared up-front in `CompilerVisitor::new` (the trampolines) and on-demand at branch targets are resolved during `finish_compilation`. + +## 5. Sandbox abstraction + +`crates/polkavm/src/sandbox.rs` defines the `Sandbox` trait (`sandbox.rs:88`) along with the helper traits `SandboxConfig`, `SandboxAddressSpace`, and `SandboxProgram`. The two concrete impls — `sandbox/linux.rs` and `sandbox/generic.rs` — differ enormously in implementation but expose the same surface: + +- `spawn(global, config, outer)` — produce a new execution context. +- `load_module(program)` — make compiled code visible in this context. +- `run()` → `InterruptKind` — execute from `next_program_counter` until the guest hits a hostcall, trap, breakpoint, or runs out of gas. +- Accessors for guest registers, memory, gas, and PC. + +Above the trait, `Instance::spawn_and_load_module` (`sandbox.rs:163`) does the actual wiring: pick the right sandbox kind based on `Config`, hand it the `CompiledModule`, and hold onto the resulting box for the lifetime of the instance. + +### 5.1 Linux sandbox — zygote + userfaultfd + +This is the fast path on Linux x86-64 and is structurally the most interesting piece of the system. + +**Zygote process.** The polkavm-zygote crate (`crates/polkavm-zygote`) builds a tiny standalone binary. The Linux sandbox embeds that binary (`sandbox/linux.rs:799`) and spawns it via `clone`/`execve`. The zygote sets up its address space — code region at `VM_ADDR_NATIVE_CODE`, guest memory, jump table, a shared `VmCtx` (from `polkavm_common::zygote`) holding atomic registers, gas, PC and a futex — then parks itself on the futex waiting for the host to drop work into shared memory. Pre-initialising the zygote once amortises page-table / mmap setup across many spawns. + +**Per-instance spawn.** `Sandbox::spawn` (`sandbox/linux.rs:1630`) clones the zygote (copy-on-write) for a new instance. Mapping a freshly compiled module is then mostly an mmap into the shared code region. + +**Userfaultfd for dynamic paging.** When dynamic paging is enabled, guest pages are not mapped up front. Instead the host registers the guest memory region with userfaultfd (`linux.rs:2071-2081`, kernel ≥ 6.8). A guest access to an unmapped page traps into the kernel, the kernel forwards the fault to the host process, the host calls `crate::compiler::on_page_fault::` (`linux.rs:2776`) to either materialise the page (`uffdio_zeropage` at `linux.rs:2868`, `uffdio_writeprotect` at `linux.rs:2884`) or convert the fault into `InterruptKind::Segfault`. The host then resumes the guest. This avoids syscalls on the hot path and gives us copy-on-write/zero-on-demand semantics for free. + +This is the reason the `unprivileged_userfaultfd` sysctl shows up in CI — without it, the worker can't register the fd and the recompiler tests fail. The interpreter doesn't use it, which is why the musttail CI job doesn't need that step. + +**Run loop.** `Sandbox::run` (`linux.rs:2123`) sets `next_native_program_counter` in `VmCtx`, flips the futex from `IDLE` to `BUSY`, and wakes the worker. The worker executes compiled code directly — no per-instruction syscalls. When the guest needs to talk to the host (hostcall, trap, gas exhaustion, page fault, step), the trampolines in the emitted code flip the futex to one of `VMCTX_FUTEX_GUEST_{ECALLI,SIGNAL,TRAP,NOT_ENOUGH_GAS,STEP,PAGEFAULT}` and park (`linux.rs:2695-2752`). The host's `run` loop wakes, inspects the state and returns the corresponding `InterruptKind`. `VMCTX_FUTEX_LONGJUMP` is used to forcibly abort the guest when the host wants to bail out. + +### 5.2 Generic sandbox — same-process signal handling + +`sandbox/generic.rs` is the portable fallback (macOS, FreeBSD, Windows in some configurations). It does not fork a separate worker: compiled code runs in the host process, called synchronously. Guest memory sits at a fixed offset from `VmCtx` (`GUEST_MEMORY_TO_VMCTX_OFFSET = -4096`, `generic.rs:120`), reached through the dedicated memory register described above. + +Faults and traps use POSIX signals — SIGSEGV for out-of-bounds memory, SIGILL for the gas-overflow trick — caught by signal handlers that convert the context into an `InterruptKind`. This is slower than userfaultfd on Linux (signal delivery and longjmp-style recovery is expensive) but does not require any OS-specific features and is gated behind the `generic-sandbox` Cargo feature. + +## 6. Caching: `CompiledModule` and the module cache + +`CompiledModule` (`compiler.rs:62-69`) is the artefact handed to a sandbox. It carries: + +- `sandbox_program: S::Program` — the assembled bytes plus any sandbox-specific descriptors, +- `native_code_origin: u64`, +- `program_counter_to_machine_code_offset_list` (sorted) and `…_map` (HashMap) — bidirectional guest-PC ↔ native-offset for resumption, breakpoints, and exports, +- `gas_metering_stub_offsets: Vec` — where every per-block gas stub lives, used by the signal/page-fault recovery path to read the block's cost and update the gas counter. + +`crates/polkavm/src/module_cache.rs` keeps two tiers — an active `BTreeMap>` of modules still alive somewhere, plus an LRU of evictable ones. The cache key is hashed from `ModuleConfig` plus the blob's content. `Module::from_blob` consults the cache before compiling (`api.rs:450-455`), which is the reason loading the same blob twice is effectively free. + +## 7. Gas metering, end to end + +Three pieces collaborate: + +1. **Compile-time accounting.** `gas_visitor: G` is called for every instruction. `G` is either a cheap per-instruction adder (`GasVisitor`) or the full `Simulator` cost model. +2. **Stub emission.** At each basic-block start, `emit_gas_metering_stub` writes a `sub qword [vmctx+gas_offset], i32::MAX`. The literal is a placeholder; once the block is closed and `gas_visitor.take_block_cost()` is called, the real cost is patched into the stub via the offsets recorded in `gas_metering_stub_offsets`. +3. **Run-time recovery.** When the subtraction underflows, the synchronous-mode stub jumps into the illegal-opcode trick, the kernel delivers SIGILL (generic) or the userfaultfd / signal pipe (Linux) tells the host. `on_page_fault` / `on_signal_trap` look up the stub's offset, restore the gas counter (the subtraction is unwound), and return `InterruptKind::NotEnoughGas` to the caller. + +The reason metering lives at block granularity rather than per instruction is that the cost of the subtraction + check is amortised across an entire straight-line block, and the stub itself is only ~10 bytes. The recompiler's per-block model is the same one the interpreter exposes via its block cache, so gas accounting is consistent across backends. + +## 8. Where to start reading the code + +If you want to follow a single guest instruction all the way through: + +1. `api.rs:403` `Module::from_blob` → `api.rs:525` `compile_module!` +2. `compiler.rs:71` `CompilerVisitor` and its `ParsingVisitor` impl around `compiler.rs:614` +3. `compiler/amd64.rs:1530` `add_generic` (or pick any other handler) — see the assembler API in action +4. `compiler.rs:308` `finish_compilation` — see how labels resolve and gas costs get patched +5. `sandbox.rs:163` `Instance::spawn_and_load_module` → `sandbox/linux.rs:1630` `Sandbox::spawn` → `sandbox/linux.rs:2123` `Sandbox::run` +6. For page-fault paths: `sandbox/linux.rs:2752` (`VMCTX_FUTEX_GUEST_PAGEFAULT` handling) → `crate::compiler::on_page_fault`. + +Those six stops cover the whole pipeline from "host hands us bytes" to "guest runs and yields a result". + +## 9. Testing at 4K page size on macOS + +macOS on Apple Silicon runs a **16K** page size natively and it cannot be changed +(the constraint is below the OS). So on a Mac the recompiler's paging / `mprotect` +boundary logic — guard pages, dynamic paging, page-fault resume — is only ever +exercised at 16K granularity, even though the silicon's MMU does support 4K. + +To test at a real 4K page size, run the recompiler inside a Linux guest under +Apple's Virtualization.framework. The guest runs a 4K-page kernel on the real +hardware MMU (not software emulation) at near-native speed, so CPU-bound guest +execution is indistinguishable from bare metal — only I/O boundaries pay VM cost. + +``` +ci/jobs/build-and-test-macos-4k-lima.sh # default recompiler tests +ci/jobs/build-and-test-macos-4k-lima.sh # custom filter +``` + +The script boots a dedicated `vz` VM via [Lima](https://lima-vm.io) (`brew install +lima`), provisions the toolchain once, asserts the guest is genuinely 4K, and runs +the generic-sandbox compiler tests against the working tree (build output is +redirected to the guest's local disk since the repo is a read-only mount). +Tear down with `limactl stop polkavm-4k && limactl delete polkavm-4k`. diff --git a/ci/hypervisor/hypervisor.entitlements b/ci/hypervisor/hypervisor.entitlements new file mode 100644 index 000000000..f57e94e8c --- /dev/null +++ b/ci/hypervisor/hypervisor.entitlements @@ -0,0 +1,5 @@ + + + + com.apple.security.hypervisor + diff --git a/ci/hypervisor/sign-and-run.sh b/ci/hypervisor/sign-and-run.sh new file mode 100755 index 000000000..ea6eae6c2 --- /dev/null +++ b/ci/hypervisor/sign-and-run.sh @@ -0,0 +1,7 @@ +#!/bin/sh +# Cargo test-runner: ad-hoc codesign the test binary with the hypervisor +# entitlement, then run it. Lets `cargo test` exercise the Hypervisor sandbox. +set -e +bin="$1"; shift +codesign -s - --entitlements "$(dirname "$0")/hypervisor.entitlements" --force "$bin" >/dev/null 2>&1 || true +exec "$bin" "$@" diff --git a/ci/jobs/build-and-test-linux-aarch64.sh b/ci/jobs/build-and-test-linux-aarch64.sh new file mode 100755 index 000000000..732866a0f --- /dev/null +++ b/ci/jobs/build-and-test-linux-aarch64.sh @@ -0,0 +1,21 @@ +#!/bin/bash + +# AArch64 recompiler tests via the generic sandbox. Unlike macOS (16K pages) this host has a 4K +# native page size, so the paging/protection boundary logic is exercised at the production granule. + +set -euo pipefail +cd -- "$(dirname -- "${BASH_SOURCE[0]}")" +cd ../.. + +echo ">> getconf PAGESIZE" +getconf PAGESIZE + +echo ">> cargo test (generic-sandbox)" +cargo test --features generic-sandbox -p polkavm + +echo ">> cargo test (assembler)" +cargo test -p polkavm-assembler --features alloc + +echo ">> cargo run (examples, compiler, generic)" +POLKAVM_TRACE_EXECUTION=1 POLKAVM_ALLOW_INSECURE=1 POLKAVM_BACKEND=compiler POLKAVM_SANDBOX=generic \ + cargo run -p hello-world-host diff --git a/ci/jobs/build-and-test-macos-4k-lima.sh b/ci/jobs/build-and-test-macos-4k-lima.sh new file mode 100755 index 000000000..6631b2add --- /dev/null +++ b/ci/jobs/build-and-test-macos-4k-lima.sh @@ -0,0 +1,68 @@ +#!/bin/bash +# Run the AArch64 recompiler (generic-sandbox) tests on a REAL 4K page size. +# +# macOS on Apple Silicon uses 16K pages natively, so the recompiler's paging / +# mprotect logic is never exercised at 4K on a Mac. A Linux guest under Apple's +# Virtualization.framework (Lima's `vz` backend) runs a 4K-page kernel on the +# real hardware MMU at near-native speed. This boots such a guest and runs the +# generic-sandbox compiler tests inside it against the working tree. +# +# Usage: +# ci/jobs/build-and-test-macos-4k-lima.sh # default recompiler tests +# ci/jobs/build-and-test-macos-4k-lima.sh +# +# Env overrides: POLKAVM_LIMA_VM, POLKAVM_LIMA_CPUS, POLKAVM_LIMA_MEM +# Teardown: limactl stop polkavm-4k && limactl delete polkavm-4k + +set -euo pipefail +cd -- "$(dirname -- "${BASH_SOURCE[0]}")" +cd ../.. +REPO="$(pwd)" + +VM="${POLKAVM_LIMA_VM:-polkavm-4k}" +CPUS="${POLKAVM_LIMA_CPUS:-8}" +MEM="${POLKAVM_LIMA_MEM:-12}" + +command -v limactl >/dev/null || { echo "!! lima not installed. Install with: brew install lima"; exit 1; } + +sh() { limactl shell "$VM" -- bash -lc "$1"; } + +# Create the VM once. vz => real hardware MMU (not TCG emulation); the stock +# Ubuntu arm64 kernel is built for 4K pages, which is exactly what we want. +if ! limactl list -q 2>/dev/null | grep -qx "$VM"; then + echo ">> creating Lima VM '$VM' (vz backend, 4K guest, ${CPUS} cpus / ${MEM} GiB)" + limactl start --vm-type=vz --cpus="$CPUS" --memory="$MEM" --name="$VM" template://ubuntu --tty=false +else + echo ">> starting Lima VM '$VM'" + limactl start "$VM" --tty=false +fi + +# Fail loudly if the guest is not actually 4K (e.g. a fallback to a 16K kernel). +PS="$(sh 'getconf PAGE_SIZE')" +echo ">> guest page size: $PS" +[ "$PS" = "4096" ] || { echo "!! guest page size is $PS, expected 4096 — aborting"; exit 1; } + +# Provision the toolchain and native build deps once (idempotent marker file). +if ! sh 'test -f "$HOME/.polkavm-4k-provisioned"'; then + echo ">> provisioning guest (apt deps + rustup)" + sh 'sudo apt-get update -qq && sudo apt-get install -y -qq gcc g++ make cmake clang pkg-config libssl-dev perl curl' + sh 'command -v cargo >/dev/null || curl --proto "=https" --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal' + sh 'touch "$HOME/.polkavm-4k-provisioned"' +fi + +# The repo is a read-only virtiofs mount inside the guest, so send cargo's build +# output to the guest's own writable disk. +# +# memset_with_dynamic_paging is skipped on macOS (16K) but PASSES at 4K, so we run +# it here. memset_basic is a pre-existing AArch64-recompiler gas-metering quirk +# (skipped on macOS too), unrelated to page size. +TEST_ARGS="${*:-tests::compiler_generic_ --skip tests::compiler_generic_memset_basic}" + +echo ">> cargo test (generic-sandbox) at 4K page size" +sh "source \$HOME/.cargo/env + cd '$REPO' + export CARGO_TARGET_DIR=\$HOME/polkavm-target-4k + echo \" toolchain: \$(rustc --version) | page size: \$(getconf PAGE_SIZE)\" + cargo test --locked --features generic-sandbox -p polkavm -- $TEST_ARGS" + +echo ">> done (4K page size validated)" diff --git a/ci/jobs/build-and-test-macos-hypervisor.sh b/ci/jobs/build-and-test-macos-hypervisor.sh new file mode 100755 index 000000000..0ce06c3d9 --- /dev/null +++ b/ci/jobs/build-and-test-macos-hypervisor.sh @@ -0,0 +1,34 @@ +#!/bin/bash + +# Hypervisor-sandbox tests. `hv_vm_create` needs the `com.apple.security.hypervisor` entitlement, +# which cargo's test binaries don't carry, so they are ad-hoc codesigned by a cargo test runner. +# +# This needs a real Apple Silicon host: GitHub's macOS runners are themselves VMs and cannot nest +# virtualization, so this job only works on self-hosted hardware. + +set -euo pipefail +cd -- "$(dirname -- "${BASH_SOURCE[0]}")" +cd ../.. + +if [ "$(uname -sm)" != "Darwin arm64" ]; then + echo ">> skipping: the hypervisor sandbox needs macOS on Apple Silicon (got '$(uname -sm)')" + exit 0 +fi + +export CARGO_TARGET_AARCH64_APPLE_DARWIN_RUNNER="$(pwd)/ci/hypervisor/sign-and-run.sh" + +echo ">> cargo test (hypervisor-sandbox only)" +cargo test --target aarch64-apple-darwin --features hypervisor-sandbox -p polkavm -- tests::aarch64_hypervisor + +echo ">> cargo test (hypervisor-sandbox + generic-sandbox)" +cargo test --target aarch64-apple-darwin --features generic-sandbox,hypervisor-sandbox -p polkavm + +echo ">> cargo test (AArch64 backend corpus routed through the hypervisor)" +POLKAVM_TEST_HYPERVISOR=1 cargo test --target aarch64-apple-darwin \ + --features generic-sandbox,hypervisor-sandbox -p polkavm -- tests::aarch64_backend + +# The whole matrix at a real 4K guest granule. Serial: one VM per process. +echo ">> cargo test (full matrix through the hypervisor, 4K guest pages)" +POLKAVM_TEST_HYPERVISOR=1 cargo test --target aarch64-apple-darwin \ + --features generic-sandbox,hypervisor-sandbox -p polkavm --lib -- \ + tests::compiler_hypervisor_ --test-threads=1 diff --git a/ci/jobs/build-and-test-macos.sh b/ci/jobs/build-and-test-macos.sh index 111a48290..55427ef29 100755 --- a/ci/jobs/build-and-test-macos.sh +++ b/ci/jobs/build-and-test-macos.sh @@ -16,7 +16,4 @@ echo ">> cargo run (examples, interpreter, aarch64-apple-darwin)" POLKAVM_TRACE_EXECUTION=1 POLKAVM_ALLOW_INSECURE=1 POLKAVM_BACKEND=interpreter cargo run --target=aarch64-apple-darwin -p hello-world-host echo ">> cargo test (generic-sandbox)" -cargo test --features generic-sandbox -p polkavm -- \ - tests::compiler_generic_ \ - --skip tests::compiler_generic_memset_basic \ - --skip tests::compiler_generic_memset_with_dynamic_paging +cargo test --features generic-sandbox -p polkavm -- tests::compiler_generic_ tests::tracing_generic_ diff --git a/ci/jobs/lib/build-and-test.sh b/ci/jobs/lib/build-and-test.sh index 14be52e68..91c28cdb4 100755 --- a/ci/jobs/lib/build-and-test.sh +++ b/ci/jobs/lib/build-and-test.sh @@ -8,7 +8,8 @@ PROFILE="${1:?usage: build-and-test.sh }" echo ">> cargo test (main crates, $PROFILE)" cargo test --profile $PROFILE -p polkavm -cargo test --profile $PROFILE -p polkavm-assembler +# `alloc` gates the assembler's own tests; without it the run is empty. +cargo test --profile $PROFILE -p polkavm-assembler --features alloc cargo test --profile $PROFILE -p polkavm-common cargo test --profile $PROFILE -p polkavm-common --all-features cargo test --profile $PROFILE -p polkavm-derive diff --git a/crates/polkavm-assembler/src/aarch64.rs b/crates/polkavm-assembler/src/aarch64.rs new file mode 100644 index 000000000..57192daf5 --- /dev/null +++ b/crates/polkavm-assembler/src/aarch64.rs @@ -0,0 +1,746 @@ +//! AArch64 (A64) instruction encoder; ARM counterpart to [`crate::amd64`]. +//! Each builder emits one little-endian `u32`; branches carry a fixup for the shared assembler to patch. + +#![allow(non_camel_case_types)] + +use crate::misc::{EncodeFlags, FixupKind, InstBuf, InstructionT, Label}; + +/// A GPR (`x0`..`x30`) or the zero register (`xzr`, encoding `31`). `sp` (also `31`) is never emitted here. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum Reg { + X0 = 0, + X1 = 1, + X2 = 2, + X3 = 3, + X4 = 4, + X5 = 5, + X6 = 6, + X7 = 7, + X8 = 8, + X9 = 9, + X10 = 10, + X11 = 11, + X12 = 12, + X13 = 13, + X14 = 14, + X15 = 15, + X16 = 16, + X17 = 17, + X18 = 18, + X19 = 19, + X20 = 20, + X21 = 21, + X22 = 22, + X23 = 23, + X24 = 24, + X25 = 25, + X26 = 26, + X27 = 27, + X28 = 28, + X29 = 29, + X30 = 30, + XZR = 31, +} + +impl Reg { + #[inline] + pub const fn idx(self) -> u32 { + self as u32 + } + + #[inline] + pub const fn equals(self, other: Self) -> bool { + (self as u32) == (other as u32) + } +} + +impl core::fmt::Display for Reg { + fn fmt(&self, fmt: &mut core::fmt::Formatter) -> core::fmt::Result { + if matches!(self, Reg::XZR) { + fmt.write_str("xzr") + } else { + fmt.write_fmt(core::format_args!("x{}", self.idx())) + } + } +} + +/// Operation width: `W` is 32-bit, `X` is 64-bit. Selects the `sf` bit. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum RegSize { + W = 0, + X = 1, +} + +impl RegSize { + #[inline] + pub const fn sf(self) -> u32 { + self as u32 + } +} + +/// Memory access width for loads and stores. The value is the A64 `size` field. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum Size { + U8 = 0, + U16 = 1, + U32 = 2, + U64 = 3, +} + +impl Size { + #[inline] + pub const fn raw(self) -> u32 { + self as u32 + } +} + +/// Index extension for register-offset loads and stores (the A64 `option` field). +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum Index { + /// 64-bit index, no extension. + Lsl = 0b011, + /// 32-bit index, zero-extended — what a wrapped guest address needs. + Uxtw = 0b010, +} + +impl Index { + #[inline] + pub const fn raw(self) -> u32 { + self as u32 + } +} + +/// An immediate encodable by the logical (bitmask) instructions, as the `N:immr:imms` fields. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct LogicalImm(u32); + +impl LogicalImm { + /// Position of the fields within the instruction word: N at 22, immr at 21:16, imms at 15:10. + #[inline] + pub const fn bits(self) -> u32 { + self.0 + } +} + +const fn is_mask(value: u64) -> bool { + value != 0 && value.wrapping_add(1) & value == 0 +} + +const fn is_shifted_mask(value: u64) -> bool { + value != 0 && is_mask(value.wrapping_sub(1) | value) +} + +/// Try to encode `value` as a logical (bitmask) immediate: a rotated run of ones, replicated across +/// the register. Ported from LLVM's `processLogicalImmediate`; `None` when no encoding exists. +pub const fn logical_imm(value: u64, size: RegSize) -> Option { + let width = match size { + RegSize::W => 32, + RegSize::X => 64, + }; + if value == 0 || value == u64::MAX { + return None; + } + if width != 64 && (value >> width != 0 || value == u64::MAX >> (64 - width)) { + return None; + } + + // Narrow down to the smallest element that repeats to fill the register. + let mut element = width; + loop { + element /= 2; + let mask = (1u64 << element) - 1; + if value & mask != (value >> element) & mask { + element *= 2; + break; + } + if element <= 2 { + break; + } + } + + // Rotate that element so it reads 0^m 1^n, and record how far we had to go. + let mask = u64::MAX >> (64 - element); + let mut value = value & mask; + let rotation; + let ones; + if is_shifted_mask(value) { + rotation = value.trailing_zeros(); + ones = (value >> rotation).trailing_ones(); + } else { + value |= !mask; + if !is_shifted_mask(!value) { + return None; + } + let leading_ones = value.leading_ones(); + rotation = 64 - leading_ones; + ones = leading_ones + value.trailing_ones() - (64 - element); + } + + let immr = (element - rotation) & (element - 1); + // Zeroes below the element's top bit, ones above it, then the run length in the low bits. + let imms = (!(element - 1) << 1) | (ones - 1); + let n = ((imms >> 6) & 1) ^ 1; + Some(LogicalImm((n << 22) | ((immr & 0x3f) << 16) | ((imms & 0x3f) << 10))) +} + +/// Condition codes for `B.cond` and conditional selects. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum Cond { + Eq = 0, + Ne = 1, + Cs = 2, // unsigned >= (HS) + Cc = 3, // unsigned < (LO) + Mi = 4, + Pl = 5, + Vs = 6, + Vc = 7, + Hi = 8, // unsigned > + Ls = 9, // unsigned <= + Ge = 10, + Lt = 11, + Gt = 12, + Le = 13, + Al = 14, +} + +impl Cond { + #[inline] + pub const fn raw(self) -> u32 { + self as u32 + } + + /// The inverted condition (e.g. used to branch *over* a long unconditional jump). + #[inline] + pub const fn invert(self) -> Cond { + match self { + Cond::Eq => Cond::Ne, + Cond::Ne => Cond::Eq, + Cond::Cs => Cond::Cc, + Cond::Cc => Cond::Cs, + Cond::Mi => Cond::Pl, + Cond::Pl => Cond::Mi, + Cond::Vs => Cond::Vc, + Cond::Vc => Cond::Vs, + Cond::Hi => Cond::Ls, + Cond::Ls => Cond::Hi, + Cond::Ge => Cond::Lt, + Cond::Lt => Cond::Ge, + Cond::Gt => Cond::Le, + Cond::Le => Cond::Gt, + Cond::Al => Cond::Al, + } + } +} + +impl core::fmt::Display for Cond { + fn fmt(&self, fmt: &mut core::fmt::Formatter) -> core::fmt::Result { + let name = match self { + Cond::Eq => "eq", + Cond::Ne => "ne", + Cond::Cs => "cs", + Cond::Cc => "cc", + Cond::Mi => "mi", + Cond::Pl => "pl", + Cond::Vs => "vs", + Cond::Vc => "vc", + Cond::Hi => "hi", + Cond::Ls => "ls", + Cond::Ge => "ge", + Cond::Lt => "lt", + Cond::Gt => "gt", + Cond::Le => "le", + Cond::Al => "al", + }; + fmt.write_str(name) + } +} + +/// Logical shift amount for shifted-register data-processing instructions. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub enum Shift { + Lsl = 0, + Lsr = 1, + Asr = 2, + Ror = 3, +} + +impl Shift { + #[inline] + pub const fn raw(self) -> u32 { + self as u32 + } +} + +/// Generates an instruction type with `encode`/`fixup`/`Display` and a constructor; `encode` yields the 32-bit word. +macro_rules! a64_inst { + ($( + $name:ident ( $($an:ident : $at:ty),* $(,)? ) => $enc:expr $(, fixup = $fix:expr)? ; + )+) => { + pub(crate) mod types { + use super::*; + $( + #[derive(Copy, Clone, PartialEq, Eq, Debug)] + pub struct $name { $(pub $an: $at),* } + + impl core::fmt::Display for $name { + fn fmt(&self, fmt: &mut core::fmt::Formatter) -> core::fmt::Result { + // Trace-only; the Debug form is good enough for disassembly logging. + fmt.write_fmt(core::format_args!("{:?}", self)) + } + } + + impl InstructionT for $name { + #[inline] + fn encode(self, _flags: EncodeFlags) -> InstBuf { + #[allow(unused_variables)] + let $name { $($an),* } = self; + let word: u32 = $enc; + let mut buf = InstBuf::new(); + buf.append_packed_bytes(word, 32); + buf + } + + #[inline] + #[allow(unused_variables)] + fn fixup(self, _flags: EncodeFlags) -> Option<(Label, FixupKind)> { + let $name { $($an),* } = self; + a64_inst!(@fixup $($fix)?) + } + } + )+ + } + + $( + #[inline] + pub fn $name($($an: $at),*) -> types::$name { + types::$name { $($an),* } + } + )+ + }; + + (@fixup) => { None }; + (@fixup $e:expr) => { $e }; +} + +a64_inst! { + // ---- Move wide immediate (MOVN/MOVZ/MOVK) ---- + // sf | opc(2) | 100101 | hw(2) | imm16 | Rd + movn(size: RegSize, rd: Reg, imm16: u16, hw: u32) => + (size.sf() << 31) | (0b100101 << 23) | (hw << 21) | ((u32::from(imm16)) << 5) | rd.idx(); + movz(size: RegSize, rd: Reg, imm16: u16, hw: u32) => + (size.sf() << 31) | (0b10 << 29) | (0b100101 << 23) | (hw << 21) | ((u32::from(imm16)) << 5) | rd.idx(); + movk(size: RegSize, rd: Reg, imm16: u16, hw: u32) => + (size.sf() << 31) | (0b11 << 29) | (0b100101 << 23) | (hw << 21) | ((u32::from(imm16)) << 5) | rd.idx(); + + // ---- Add/Subtract (immediate) ---- + // sf | op | S | 100010 | sh | imm12 | Rn | Rd + add_imm(size: RegSize, rd: Reg, rn: Reg, imm12: u32, shift12: bool) => + (size.sf() << 31) | (0b100010 << 23) | ((u32::from(shift12)) << 22) | ((imm12 & 0xfff) << 10) | (rn.idx() << 5) | rd.idx(); + sub_imm(size: RegSize, rd: Reg, rn: Reg, imm12: u32, shift12: bool) => + (size.sf() << 31) | (1 << 30) | (0b100010 << 23) | ((u32::from(shift12)) << 22) | ((imm12 & 0xfff) << 10) | (rn.idx() << 5) | rd.idx(); + subs_imm(size: RegSize, rd: Reg, rn: Reg, imm12: u32, shift12: bool) => + (size.sf() << 31) | (1 << 30) | (1 << 29) | (0b100010 << 23) | ((u32::from(shift12)) << 22) | ((imm12 & 0xfff) << 10) | (rn.idx() << 5) | rd.idx(); + + // ---- Add/Subtract (shifted register), shift amount 0 ---- + // sf | op | S | 01011 | shift(2) | 0 | Rm | imm6 | Rn | Rd + add_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b01011 << 24) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + sub_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (1 << 30) | (0b01011 << 24) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + subs_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (1 << 30) | (1 << 29) | (0b01011 << 24) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + + // ---- Logical (shifted register), shift amount 0, N=0 ---- + // sf | opc(2) | 01010 | shift(2) | N | Rm | imm6 | Rn | Rd + and_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b01010 << 24) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + orr_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b01 << 29) | (0b01010 << 24) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + eor_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b10 << 29) | (0b01010 << 24) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + + // ---- Load/Store register (unsigned immediate offset), offset scaled by access size ---- + // size(2) | 111 | 0 | 01 | opc(2) | imm12 | Rn | Rt + str_imm(size: Size, rt: Reg, rn: Reg, imm12_scaled: u32) => + (size.raw() << 30) | (0b111 << 27) | (0b01 << 24) | ((imm12_scaled & 0xfff) << 10) | (rn.idx() << 5) | rt.idx(); + ldr_imm(size: Size, rt: Reg, rn: Reg, imm12_scaled: u32) => + (size.raw() << 30) | (0b111 << 27) | (0b01 << 24) | (0b01 << 22) | ((imm12_scaled & 0xfff) << 10) | (rn.idx() << 5) | rt.idx(); + // Sign-extending load into a 64-bit register (opc = 10). + ldrs64_imm(size: Size, rt: Reg, rn: Reg, imm12_scaled: u32) => + (size.raw() << 30) | (0b111 << 27) | (0b01 << 24) | (0b10 << 22) | ((imm12_scaled & 0xfff) << 10) | (rn.idx() << 5) | rt.idx(); + + // ---- Load/Store register (unscaled signed 9-bit immediate): LDUR/STUR ---- + // size(2) | 111 | 0 | 00 | opc(2) | 0 | imm9 | 00 | Rn | Rt + stur(size: Size, rt: Reg, rn: Reg, imm9: i32) => + (size.raw() << 30) | (0b111 << 27) | (((imm9 as u32) & 0x1ff) << 12) | (rn.idx() << 5) | rt.idx(); + ldur(size: Size, rt: Reg, rn: Reg, imm9: i32) => + (size.raw() << 30) | (0b111 << 27) | (0b01 << 22) | (((imm9 as u32) & 0x1ff) << 12) | (rn.idx() << 5) | rt.idx(); + + // ---- Unconditional branch (immediate) ---- + // op | 00101 | imm26 + b(label: Label) => 0b000101 << 26, fixup = Some((label, FixupKind::aarch64_branch(0, 26))); + bl(label: Label) => 0b100101 << 26, fixup = Some((label, FixupKind::aarch64_branch(0, 26))); + + // ---- Conditional branch (immediate) ---- + // 0101010 0 | imm19 | 0 | cond + b_cond(cond: Cond, label: Label) => (0b01010100 << 24) | cond.raw(), fixup = Some((label, FixupKind::aarch64_branch(5, 19))); + + // ---- Compare and branch ---- + // sf | 011010 | op | imm19 | Rt + cbz(size: RegSize, rt: Reg, label: Label) => + (size.sf() << 31) | (0b011010 << 25) | rt.idx(), fixup = Some((label, FixupKind::aarch64_branch(5, 19))); + cbnz(size: RegSize, rt: Reg, label: Label) => + (size.sf() << 31) | (0b011010 << 25) | (1 << 24) | rt.idx(), fixup = Some((label, FixupKind::aarch64_branch(5, 19))); + + // ---- PC-relative address ---- + // 0 immlo(2) 10000 immhi(19) Rd ; 21-bit signed byte offset relative to this instruction. + adr(rd: Reg, label: Label) => 0x1000_0000 | rd.idx(), fixup = Some((label, FixupKind::aarch64_adr())); + // 1 immlo(2) 10000 immhi(19) Rd ; 21-bit signed 4 KiB-page offset (±4 GiB). Yields the base of + // the target's page, so the caller must add the low 12 bits (or know the target is 4K-aligned). + adrp(rd: Reg, label: Label) => 0x9000_0000 | rd.idx(), fixup = Some((label, FixupKind::aarch64_adrp())); + + // ---- Unconditional branch (register) ---- + br(rn: Reg) => 0xD61F0000 | (rn.idx() << 5); + blr(rn: Reg) => 0xD63F0000 | (rn.idx() << 5); + ret(rn: Reg) => 0xD65F0000 | (rn.idx() << 5); + + // ---- Logical (shifted register), inverted forms (N=1), shift amount 0 ---- + bic_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b01010 << 24) | (1 << 21) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + orn_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b01 << 29) | (0b01010 << 24) | (1 << 21) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + eon_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b10 << 29) | (0b01010 << 24) | (1 << 21) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + ands_reg(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b11 << 29) | (0b01010 << 24) | (rm.idx() << 16) | (rn.idx() << 5) | rd.idx(); + + // ---- Multiply (3-source) ---- + // sf 00 11011 000 Rm o0 Ra Rn Rd (MADD o0=0, MSUB o0=1) + madd(size: RegSize, rd: Reg, rn: Reg, rm: Reg, ra: Reg) => + (size.sf() << 31) | (0b0011011 << 24) | (rm.idx() << 16) | (ra.idx() << 10) | (rn.idx() << 5) | rd.idx(); + msub(size: RegSize, rd: Reg, rn: Reg, rm: Reg, ra: Reg) => + (size.sf() << 31) | (0b0011011 << 24) | (rm.idx() << 16) | (1 << 15) | (ra.idx() << 10) | (rn.idx() << 5) | rd.idx(); + // SMULH / UMULH (64-bit high half), Ra = xzr + smulh(rd: Reg, rn: Reg, rm: Reg) => + 0x9B40_0000 | (rm.idx() << 16) | (0b11111 << 10) | (rn.idx() << 5) | rd.idx(); + umulh(rd: Reg, rn: Reg, rm: Reg) => + 0x9BC0_0000 | (rm.idx() << 16) | (0b11111 << 10) | (rn.idx() << 5) | rd.idx(); + + // ---- Data-processing (2-source): divide and variable shift ---- + // sf 0 0 11010110 Rm opcode(6) Rn Rd + udiv(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b0011010110 << 21) | (rm.idx() << 16) | (0b000010 << 10) | (rn.idx() << 5) | rd.idx(); + sdiv(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b0011010110 << 21) | (rm.idx() << 16) | (0b000011 << 10) | (rn.idx() << 5) | rd.idx(); + lslv(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b0011010110 << 21) | (rm.idx() << 16) | (0b001000 << 10) | (rn.idx() << 5) | rd.idx(); + lsrv(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b0011010110 << 21) | (rm.idx() << 16) | (0b001001 << 10) | (rn.idx() << 5) | rd.idx(); + asrv(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b0011010110 << 21) | (rm.idx() << 16) | (0b001010 << 10) | (rn.idx() << 5) | rd.idx(); + rorv(size: RegSize, rd: Reg, rn: Reg, rm: Reg) => + (size.sf() << 31) | (0b0011010110 << 21) | (rm.idx() << 16) | (0b001011 << 10) | (rn.idx() << 5) | rd.idx(); + + // ---- Data-processing (1-source): bit counting / reversal ---- + // Base opcodes (32-bit); the `sf` bit selects the 64-bit form. + rbit(size: RegSize, rd: Reg, rn: Reg) => + 0x5AC0_0000 | (size.sf() << 31) | (rn.idx() << 5) | rd.idx(); + clz(size: RegSize, rd: Reg, rn: Reg) => + 0x5AC0_1000 | (size.sf() << 31) | (rn.idx() << 5) | rd.idx(); + rev16(size: RegSize, rd: Reg, rn: Reg) => + 0x5AC0_0400 | (size.sf() << 31) | (rn.idx() << 5) | rd.idx(); + // REV: 32-bit form reverses the 4 bytes of a word; 64-bit form (opc=0b11) reverses all 8 bytes. + rev32_word(rd: Reg, rn: Reg) => 0x5AC0_0800 | (rn.idx() << 5) | rd.idx(); + rev64(rd: Reg, rn: Reg) => 0xDAC0_0C00 | (rn.idx() << 5) | rd.idx(); + + // ---- Bitfield move (SBFM / UBFM): used for shifts-by-immediate and sign/zero extension ---- + // sf opc(2) 100110 N immr(6) imms(6) Rn Rd ; for 64-bit N=1, for 32-bit N=0 + sbfm(size: RegSize, rd: Reg, rn: Reg, immr: u32, imms: u32) => + (size.sf() << 31) | (0b100110 << 23) | (size.sf() << 22) | ((immr & 0x3f) << 16) | ((imms & 0x3f) << 10) | (rn.idx() << 5) | rd.idx(); + ubfm(size: RegSize, rd: Reg, rn: Reg, immr: u32, imms: u32) => + (size.sf() << 31) | (0b10 << 29) | (0b100110 << 23) | (size.sf() << 22) | ((immr & 0x3f) << 16) | ((imms & 0x3f) << 10) | (rn.idx() << 5) | rd.idx(); + + // ---- Extract register (EXTR); `extr rd, rn, rn, #n` is `ror rd, rn, #n` ---- + // sf 00 100111 N 0 Rm imms(6) Rn Rd + extr(size: RegSize, rd: Reg, rn: Reg, rm: Reg, lsb: u32) => + (size.sf() << 31) | (0b100111 << 23) | (size.sf() << 22) | (rm.idx() << 16) | ((lsb & 0x3f) << 10) | (rn.idx() << 5) | rd.idx(); + + // ---- Logical (immediate) ---- + // sf opc(2) 100100 N immr(6) imms(6) Rn Rd ; the immediate is a bitmask pattern, see `logical_imm`. + and_imm(size: RegSize, rd: Reg, rn: Reg, imm: LogicalImm) => + (size.sf() << 31) | (0b100100 << 23) | imm.bits() | (rn.idx() << 5) | rd.idx(); + orr_imm(size: RegSize, rd: Reg, rn: Reg, imm: LogicalImm) => + (size.sf() << 31) | (0b01 << 29) | (0b100100 << 23) | imm.bits() | (rn.idx() << 5) | rd.idx(); + eor_imm(size: RegSize, rd: Reg, rn: Reg, imm: LogicalImm) => + (size.sf() << 31) | (0b10 << 29) | (0b100100 << 23) | imm.bits() | (rn.idx() << 5) | rd.idx(); + + // ---- Conditional select ---- + // sf 0 0 11010100 Rm cond op2(2) Rn Rd + csel(size: RegSize, rd: Reg, rn: Reg, rm: Reg, cond: Cond) => + (size.sf() << 31) | (0b0011010100 << 21) | (rm.idx() << 16) | (cond.raw() << 12) | (rn.idx() << 5) | rd.idx(); + csinc(size: RegSize, rd: Reg, rn: Reg, rm: Reg, cond: Cond) => + (size.sf() << 31) | (0b0011010100 << 21) | (rm.idx() << 16) | (cond.raw() << 12) | (0b01 << 10) | (rn.idx() << 5) | rd.idx(); + // CSINV: rd = cond ? rn : ~rm + csinv(size: RegSize, rd: Reg, rn: Reg, rm: Reg, cond: Cond) => + 0x5A80_0000 | (size.sf() << 31) | (rm.idx() << 16) | (cond.raw() << 12) | (rn.idx() << 5) | rd.idx(); + + // ---- Load/Store register (register offset), scale S = 0 ---- + // size 111 0 00 opc 1 Rm option(3) S=0 10 Rn Rt + str_reg(size: Size, rt: Reg, rn: Reg, rm: Reg, index: Index) => + (size.raw() << 30) | (0b111 << 27) | (1 << 21) | (rm.idx() << 16) | (index.raw() << 13) | (0b10 << 10) | (rn.idx() << 5) | rt.idx(); + ldr_reg(size: Size, rt: Reg, rn: Reg, rm: Reg, index: Index) => + (size.raw() << 30) | (0b111 << 27) | (0b01 << 22) | (1 << 21) | (rm.idx() << 16) | (index.raw() << 13) | (0b10 << 10) | (rn.idx() << 5) | rt.idx(); + // Sign-extending load (unsigned immediate offset) to a 32-bit register (opc = 11). + ldrs32_imm(size: Size, rt: Reg, rn: Reg, imm12_scaled: u32) => + (size.raw() << 30) | (0b111 << 27) | (0b01 << 24) | (0b11 << 22) | ((imm12_scaled & 0xfff) << 10) | (rn.idx() << 5) | rt.idx(); + // Sign-extending load (register offset) into a 32-bit register (opc = 11). + ldrs32_reg(size: Size, rt: Reg, rn: Reg, rm: Reg, index: Index) => + (size.raw() << 30) | (0b111 << 27) | (0b11 << 22) | (1 << 21) | (rm.idx() << 16) | (index.raw() << 13) | (0b10 << 10) | (rn.idx() << 5) | rt.idx(); + // Sign-extending load (register offset) into a 64-bit register (opc = 10). + ldrs64_reg(size: Size, rt: Reg, rn: Reg, rm: Reg, index: Index) => + (size.raw() << 30) | (0b111 << 27) | (0b10 << 22) | (1 << 21) | (rm.idx() << 16) | (index.raw() << 13) | (0b10 << 10) | (rn.idx() << 5) | rt.idx(); + + // ---- NEON: used for popcount (cnt). SIMD registers are passed as raw indices 0..31. ---- + fmov_gpr_to_s(sd: u32, wn: Reg) => 0x1E27_0000 | (wn.idx() << 5) | sd; // 32-bit GPR -> SIMD + fmov_gpr_to_d(dd: u32, xn: Reg) => 0x9E67_0000 | (xn.idx() << 5) | dd; // 64-bit GPR -> SIMD + fmov_s_to_gpr(wd: Reg, sn: u32) => 0x1E26_0000 | (sn << 5) | wd.idx(); // SIMD -> 32-bit GPR + cnt_8b(vd: u32, vn: u32) => 0x0E20_5800 | (vn << 5) | vd; // per-byte popcount of Vn.8B + addv_8b(bd: u32, vn: u32) => 0x0E31_B800 | (vn << 5) | bd; // sum the 8 bytes of Vn.8B + + // ---- Exceptions / hint ---- + // UDF #imm16 — permanently undefined; raises an Undefined Instruction exception (SIGILL). + udf(imm16: u16) => u32::from(imm16); + // BRK #imm16 — software breakpoint (SIGTRAP). + brk(imm16: u16) => 0xD4200000 | ((u32::from(imm16)) << 5); + nop() => 0xD503201F; +} + +/// `mov rd, rm` — encoded as `orr rd, xzr, rm`. +#[inline] +pub fn mov_reg(size: RegSize, rd: Reg, rm: Reg) -> types::orr_reg { + orr_reg(size, rd, Reg::XZR, rm) +} + +/// `cmp rn, rm` — encoded as `subs xzr, rn, rm`. +#[inline] +pub fn cmp_reg(size: RegSize, rn: Reg, rm: Reg) -> types::subs_reg { + subs_reg(size, Reg::XZR, rn, rm) +} + +/// `cmp rn, #imm12` — encoded as `subs xzr, rn, #imm12`. +#[inline] +pub fn cmp_imm(size: RegSize, rn: Reg, imm12: u32, shift12: bool) -> types::subs_imm { + subs_imm(size, Reg::XZR, rn, imm12, shift12) +} + +#[cfg(all(test, feature = "alloc"))] +mod tests { + use super::*; + + fn word(inst: T) -> u32 { + let bytes = inst.encode(EncodeFlags::default()).to_vec(); + assert_eq!(bytes.len(), 4, "every A64 instruction must be 4 bytes"); + u32::from_le_bytes([bytes[0], bytes[1], bytes[2], bytes[3]]) + } + + #[test] + fn encodings_match_known_good() { + // Cross-checked against `llvm-mc -arch=aarch64 --show-encoding`. + assert_eq!(word(movz(RegSize::X, Reg::X0, 0x1234, 0)), 0xD282_4680); + assert_eq!(word(movk(RegSize::X, Reg::X0, 0xABCD, 1)), 0xF2B5_79A0); + assert_eq!(word(movz(RegSize::W, Reg::X1, 0, 0)), 0x5280_0001); + assert_eq!(word(add_imm(RegSize::X, Reg::X0, Reg::X1, 0x10, false)), 0x9100_4020); + assert_eq!(word(sub_imm(RegSize::X, Reg::X2, Reg::X3, 1, false)), 0xD100_0462); + assert_eq!(word(add_reg(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0x8B02_0020); + assert_eq!(word(add_reg(RegSize::W, Reg::X0, Reg::X1, Reg::X2)), 0x0B02_0020); + assert_eq!(word(sub_reg(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0xCB02_0020); + assert_eq!(word(orr_reg(RegSize::X, Reg::X0, Reg::XZR, Reg::X1)), 0xAA01_03E0); // mov x0, x1 + assert_eq!(word(and_reg(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0x8A02_0020); + assert_eq!(word(eor_reg(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0xCA02_0020); + assert_eq!(word(subs_reg(RegSize::X, Reg::XZR, Reg::X1, Reg::X2)), 0xEB02_003F); // cmp x1, x2 + assert_eq!(word(ldr_imm(Size::U64, Reg::X0, Reg::X1, 1)), 0xF940_0420); // ldr x0,[x1,#8] + assert_eq!(word(str_imm(Size::U32, Reg::X0, Reg::X1, 2)), 0xB900_0820); // str w0,[x1,#8] + assert_eq!(word(ldur(Size::U64, Reg::X0, Reg::X1, -8)), 0xF85F_8020); + assert_eq!(word(br(Reg::X8)), 0xD61F_0100); + assert_eq!(word(ret(Reg::X30)), 0xD65F_03C0); + assert_eq!(word(udf(0)), 0x0000_0000); + assert_eq!(word(brk(0)), 0xD420_0000); + assert_eq!(word(nop()), 0xD503_201F); + } + + #[test] + fn extended_encodings_match_known_good() { + // mul/div/shift/bit/select/extend, cross-checked against `llvm-mc -arch=aarch64`. + assert_eq!(word(madd(RegSize::X, Reg::X0, Reg::X1, Reg::X2, Reg::XZR)), 0x9B02_7C20); // mul x0,x1,x2 + assert_eq!(word(udiv(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0x9AC2_0820); + assert_eq!(word(sdiv(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0x9AC2_0C20); + assert_eq!(word(lslv(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0x9AC2_2020); + assert_eq!(word(asrv(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0x9AC2_2820); + assert_eq!(word(smulh(Reg::X0, Reg::X1, Reg::X2)), 0x9B42_7C20); + assert_eq!(word(umulh(Reg::X0, Reg::X1, Reg::X2)), 0x9BC2_7C20); + assert_eq!(word(clz(RegSize::X, Reg::X0, Reg::X1)), 0xDAC0_1020); + assert_eq!(word(rbit(RegSize::X, Reg::X0, Reg::X1)), 0xDAC0_0020); + assert_eq!(word(csel(RegSize::X, Reg::X0, Reg::X1, Reg::X2, Cond::Eq)), 0x9A82_0020); + assert_eq!(word(csinc(RegSize::X, Reg::X0, Reg::X1, Reg::X2, Cond::Ne)), 0x9A82_1420); + assert_eq!(word(sbfm(RegSize::X, Reg::X0, Reg::X1, 0, 7)), 0x9340_1C20); // sxtb x0,x1 + assert_eq!(word(ubfm(RegSize::X, Reg::X0, Reg::X1, 1, 63)), 0xD341_FC20); // lsr x0,x1,#1 + assert_eq!(word(bic_reg(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0x8A22_0020); + assert_eq!(word(orn_reg(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0xAA22_0020); + assert_eq!(word(ands_reg(RegSize::X, Reg::X0, Reg::X1, Reg::X2)), 0xEA02_0020); + assert_eq!(word(ldr_reg(Size::U64, Reg::X0, Reg::X1, Reg::X2, Index::Lsl)), 0xF862_6820); + assert_eq!(word(str_reg(Size::U64, Reg::X0, Reg::X1, Reg::X2, Index::Lsl)), 0xF822_6820); + // Cross-checked against `clang -x assembler` on this host. + assert_eq!(word(ldr_reg(Size::U64, Reg::X0, Reg::X14, Reg::X1, Index::Uxtw)), 0xF861_49C0); + assert_eq!(word(str_reg(Size::U64, Reg::X0, Reg::X14, Reg::X1, Index::Uxtw)), 0xF821_49C0); + assert_eq!(word(ldr_reg(Size::U8, Reg::X0, Reg::X14, Reg::X1, Index::Uxtw)), 0x3861_49C0); + assert_eq!(word(ldrs64_reg(Size::U32, Reg::X0, Reg::X14, Reg::X1, Index::Uxtw)), 0xB8A1_49C0); + assert_eq!(word(ldrs64_reg(Size::U8, Reg::X0, Reg::X14, Reg::X1, Index::Uxtw)), 0x38A1_49C0); + assert_eq!(word(ldrs32_reg(Size::U8, Reg::X0, Reg::X14, Reg::X1, Index::Uxtw)), 0x38E1_49C0); + assert_eq!(word(ldrs32_reg(Size::U16, Reg::X0, Reg::X14, Reg::X1, Index::Uxtw)), 0x78E1_49C0); + assert_eq!(word(ldr_imm(Size::U64, Reg::X0, Reg::X14, 1)), 0xF940_05C0); // ldr x0,[x14,#8] + assert_eq!(word(ldr_imm(Size::U8, Reg::X0, Reg::X14, 4095)), 0x397F_FDC0); + assert_eq!(word(extr(RegSize::X, Reg::X0, Reg::X1, Reg::X1, 3)), 0x93C1_0C20); // ror x0,x1,#3 + assert_eq!(word(ubfm(RegSize::X, Reg::X0, Reg::X1, 61, 60)), 0xD37D_F020); // lsl x0,x1,#3 + assert_eq!(word(ubfm(RegSize::X, Reg::X0, Reg::X1, 3, 63)), 0xD343_FC20); // lsr x0,x1,#3 + assert_eq!(word(sbfm(RegSize::X, Reg::X0, Reg::X1, 3, 63)), 0x9343_FC20); // asr x0,x1,#3 + assert_eq!(word(ubfm(RegSize::W, Reg::X0, Reg::X1, 29, 28)), 0x531D_7020); + // lsl w0,w1,#3 + } + + #[test] + fn logical_immediates_match_known_good() { + // Cross-checked against `clang -x assembler` on this host. + let x = |value| logical_imm(value, RegSize::X).unwrap(); + assert_eq!(word(and_imm(RegSize::X, Reg::X0, Reg::X1, x(0xff))), 0x9240_1C20); + assert_eq!(word(orr_imm(RegSize::X, Reg::X0, Reg::X1, x(0xff))), 0xB240_1C20); + assert_eq!(word(eor_imm(RegSize::X, Reg::X0, Reg::X1, x(0xff))), 0xD240_1C20); + assert_eq!(word(and_imm(RegSize::X, Reg::X2, Reg::X3, x(0xffff_ffff_ffff_f000))), 0x9274_CC62); + assert_eq!(word(orr_imm(RegSize::X, Reg::X0, Reg::X1, x(0x5555_5555_5555_5555))), 0xB200_F020); + let w = logical_imm(0xff, RegSize::W).unwrap(); + assert_eq!(word(and_imm(RegSize::W, Reg::X0, Reg::X1, w)), 0x1200_1C20); + } + + #[test] + fn logical_immediates_reject_unencodable() { + assert!(logical_imm(0, RegSize::X).is_none()); + assert!(logical_imm(u64::MAX, RegSize::X).is_none()); + assert!(logical_imm(0xff, RegSize::W).is_some()); + assert!(logical_imm(0x1_0000_0000, RegSize::W).is_none()); // doesn't fit 32 bits + assert!(logical_imm(0xffff_ffff, RegSize::W).is_none()); // all-ones in W + assert!(logical_imm(0x1234_5678, RegSize::X).is_none()); // not a rotated run of ones + assert!(logical_imm(0x8000_0000_0000_0001, RegSize::X).is_some()); // wraps around + assert_eq!(word(fmov_gpr_to_s(0, Reg::X1)), 0x1E27_0020); + assert_eq!(word(fmov_gpr_to_d(0, Reg::X1)), 0x9E67_0020); + assert_eq!(word(fmov_s_to_gpr(Reg::X0, 1)), 0x1E26_0020); + assert_eq!(word(cnt_8b(0, 1)), 0x0E20_5820); + assert_eq!(word(addv_8b(0, 1)), 0x0E31_B820); + } + + #[test] + fn branch_fixups_are_patched() { + let mut asm = crate::Assembler::new(); + // Forward conditional branch over one instruction. + let target = asm.forward_declare_label(); + asm.push(b_cond(Cond::Eq, target)); + asm.push(nop()); + asm.define_label(target); + asm.push(ret(Reg::X30)); + let code = asm.finalize(); + let b0 = u32::from_le_bytes([code[0], code[1], code[2], code[3]]); + // imm19 should be +2 words (skip the b.cond itself and the nop). + let imm19 = (b0 >> 5) & 0x7ffff; + assert_eq!(imm19, 2); + assert_eq!(b0 & 0xf, Cond::Eq.raw()); + } + + #[test] + fn adr_fixup_is_patched() { + let mut asm = crate::Assembler::new(); + // adr x0, target ; nop ; nop ; target: + let target = asm.forward_declare_label(); + asm.push(adr(Reg::X0, target)); + asm.push(nop()); + asm.push(nop()); + asm.define_label(target); + let code = asm.finalize(); + let w = u32::from_le_bytes([code[0], code[1], code[2], code[3]]); + // Offset is +12 bytes (3 instructions). immlo = 12 & 3 = 0; immhi = 12 >> 2 = 3. + let immlo = (w >> 29) & 0b11; + let immhi = (w >> 5) & 0x7_ffff; + assert_eq!(immlo, 0); + assert_eq!(immhi, 3); + assert_eq!(w & 0x1f, 0); // Rd = x0 + } + + #[test] + fn adrp_fixup_is_page_relative() { + let mut asm = crate::Assembler::new(); + asm.set_origin(0x1000); // pc page = 1 + let target = asm.forward_declare_label(); + asm.push(adrp(Reg::X0, target)); + asm.resize(0x3000, 0); // target lands at 0x1000 + 0x3000 = 0x4000, page 4 + asm.define_label(target); + let code = asm.finalize(); + let w = u32::from_le_bytes([code[0], code[1], code[2], code[3]]); + let immlo = (w >> 29) & 0b11; + let immhi = (w >> 5) & 0x7_ffff; + assert_eq!((immhi << 2) | immlo, 3); // +3 pages + assert_eq!(w & 0x9f00_0000, 0x9000_0000); // ADRP + assert_eq!(w & 0x1f, 0); // Rd = x0 + } + + #[test] + fn adrp_reaches_far_beyond_adr_range() { + // The same distance would blow ADR's ±1 MiB range; ADRP encodes it as pages. + let mut asm = crate::Assembler::new(); + let target = asm.forward_declare_label(); + asm.push(adrp(Reg::X0, target)); + asm.resize(64 * 1024 * 1024, 0); + asm.define_label(target); + let code = asm.finalize(); + let w = u32::from_le_bytes([code[0], code[1], code[2], code[3]]); + let immlo = (w >> 29) & 0b11; + let immhi = (w >> 5) & 0x7_ffff; + assert_eq!((immhi << 2) | immlo, (64 * 1024 * 1024) >> 12); + } + + #[test] + fn out_of_range_fixups_are_reported_not_panicked() { + // `b` reaches ±128 MiB and `adrp` ±4 GiB; past that `try_finalize` must return an error so + // the caller can turn it into a compilation failure. + let mut asm = crate::Assembler::new(); + let target = asm.forward_declare_label(); + asm.push(b(target)); + asm.resize(256 * 1024 * 1024, 0); + asm.define_label(target); + assert_eq!(asm.try_finalize().err(), Some(crate::OutOfRange("jump"))); + + let mut asm = crate::Assembler::new(); + let target = asm.forward_declare_label(); + asm.push(adr(Reg::X0, target)); + asm.resize(2 * 1024 * 1024, 0); + asm.define_label(target); + assert_eq!(asm.try_finalize().err(), Some(crate::OutOfRange("ADR"))); + + // Within range: still fine. + let mut asm = crate::Assembler::new(); + let target = asm.forward_declare_label(); + asm.push(b(target)); + asm.resize(1024, 0); + asm.define_label(target); + assert!(asm.try_finalize().is_ok()); + } + + #[test] + fn backward_branch_is_negative() { + let mut asm = crate::Assembler::new(); + let head = asm.create_label(); + asm.push(nop()); + asm.push(b(head)); + let code = asm.finalize(); + let b1 = u32::from_le_bytes([code[4], code[5], code[6], code[7]]); + let imm26 = b1 & 0x03ff_ffff; + // -1 word in 26-bit two's complement. + assert_eq!(imm26, 0x03ff_ffff); + } +} diff --git a/crates/polkavm-assembler/src/amd64.rs b/crates/polkavm-assembler/src/amd64.rs index 15ce9a918..6953aa11e 100644 --- a/crates/polkavm-assembler/src/amd64.rs +++ b/crates/polkavm-assembler/src/amd64.rs @@ -2392,6 +2392,7 @@ mod tests { } } + #[cfg(target_arch = "x86_64")] // only the label tests below format their assertions use alloc::format; use alloc::string::String; @@ -2593,6 +2594,8 @@ mod tests { xor, } + // Label fixups are resolved by `Assembler::finalize`, which encodes for the *host* arch. + #[cfg(target_arch = "x86_64")] #[test] fn jmp_label8_infinite_loop() { use super::inst::*; @@ -2613,6 +2616,8 @@ mod tests { assert_eq!(disassembly, "00000000 0f0b ud2"); } + // Label fixups are resolved by `Assembler::finalize`, which encodes for the *host* arch. + #[cfg(target_arch = "x86_64")] #[test] fn jmp_label32_infinite_loop() { use super::inst::*; @@ -2633,6 +2638,8 @@ mod tests { assert_eq!(disassembly, "00000000 0f0b ud2\n00000002 90 nop\n00000003 0f0b ud2"); } + // Label fixups are resolved by `Assembler::finalize`, which encodes for the *host* arch. + #[cfg(target_arch = "x86_64")] #[test] fn call_label32_infinite_loop() { use super::inst::*; @@ -2653,6 +2660,8 @@ mod tests { assert_eq!(disassembly, "00000000 0f0b ud2\n00000002 90 nop\n00000003 0f0b ud2"); } + // Label fixups are resolved by `Assembler::finalize`, which encodes for the *host* arch. + #[cfg(target_arch = "x86_64")] #[test] fn jcc_label8_infinite_loop() { use super::inst::*; @@ -2680,6 +2689,8 @@ mod tests { }); } + // Label fixups are resolved by `Assembler::finalize`, which encodes for the *host* arch. + #[cfg(target_arch = "x86_64")] #[test] fn jcc_label8_jump_forward() { use super::inst::*; @@ -2700,6 +2711,8 @@ mod tests { }) } + // Label fixups are resolved by `Assembler::finalize`, which encodes for the *host* arch. + #[cfg(target_arch = "x86_64")] #[test] fn jcc_label8_jump_backward() { use super::inst::*; @@ -2720,6 +2733,8 @@ mod tests { }); } + // Label fixups are resolved by `Assembler::finalize`, which encodes for the *host* arch. + #[cfg(target_arch = "x86_64")] #[test] fn jcc_label32_jump_forward() { use super::inst::*; @@ -2752,6 +2767,8 @@ mod tests { }); } + // Label fixups are resolved by `Assembler::finalize`, which encodes for the *host* arch. + #[cfg(target_arch = "x86_64")] #[test] fn lea_rip_label_infinite_loop() { use super::inst::*; @@ -2777,6 +2794,8 @@ mod tests { }); } + // Label fixups are resolved by `Assembler::finalize`, which encodes for the *host* arch. + #[cfg(target_arch = "x86_64")] #[test] fn lea_rip_label_next_instruction() { use super::inst::*; diff --git a/crates/polkavm-assembler/src/assembler.rs b/crates/polkavm-assembler/src/assembler.rs index b0ede4d63..8424afc37 100644 --- a/crates/polkavm-assembler/src/assembler.rs +++ b/crates/polkavm-assembler/src/assembler.rs @@ -5,6 +5,8 @@ use alloc::vec::Vec; struct Fixup { target_label: Label, instruction_offset: usize, + // AArch64 instructions are always 4 bytes. + #[cfg(not(target_arch = "aarch64"))] instruction_length: u8, kind: FixupKind, } @@ -24,6 +26,16 @@ impl Default for Assembler { } } +/// A fixup target that no encoding can reach; the payload names the instruction class. +#[derive(Copy, Clone, PartialEq, Eq, Debug)] +pub struct OutOfRange(pub &'static str); + +impl core::fmt::Display for OutOfRange { + fn fmt(&self, fmt: &mut core::fmt::Formatter) -> core::fmt::Result { + fmt.write_fmt(core::format_args!("out of range {}", self.0)) + } +} + #[repr(transparent)] pub struct AssembledCode<'a>(&'a mut Assembler); @@ -230,17 +242,24 @@ impl Assembler { #[inline(always)] fn add_fixup(&mut self, instruction_offset: usize, instruction_length: usize, target_label: Label, kind: FixupKind) { debug_assert!((target_label.raw() as usize) < self.labels.len()); - debug_assert!( - (kind.offset() as usize) < instruction_length, - "instruction is {} bytes long and yet its target fixup starts at {}", - instruction_length, - kind.offset() - ); - debug_assert!((kind.length() as usize) < instruction_length); - debug_assert!((kind.offset() as usize + kind.length() as usize) <= instruction_length); + // AArch64 bit-packs the offset. + #[cfg(target_arch = "x86_64")] + { + debug_assert!( + (kind.offset() as usize) < instruction_length, + "instruction is {} bytes long and yet its target fixup starts at {}", + instruction_length, + kind.offset() + ); + debug_assert!((kind.length() as usize) < instruction_length); + debug_assert!((kind.offset() as usize + kind.length() as usize) <= instruction_length); + } + #[cfg(target_arch = "aarch64")] + let _ = instruction_length; self.fixups.push(Fixup { target_label, instruction_offset, + #[cfg(not(target_arch = "aarch64"))] instruction_length: instruction_length as u8, kind, }); @@ -306,7 +325,28 @@ impl Assembler { self } + /// Resolves every fixup, or fails when a target is out of the encodable range (which for a + /// recompiler means the input program was too big, not that the assembler is broken). + pub fn try_finalize(&mut self) -> Result { + #[cfg(target_arch = "aarch64")] + self.finalize_aarch64()?; + #[cfg(not(target_arch = "aarch64"))] + self.finalize_x86()?; + + Ok(AssembledCode(self)) + } + + /// [`Self::try_finalize`], panicking on an out-of-range fixup. pub fn finalize(&mut self) -> AssembledCode { + match self.try_finalize() { + Ok(code) => code, + Err(error) => panic!("{error}"), + } + } + + /// Resolves x86 fixups: a 1/4-byte LE displacement after the opcode, relative to the instruction end. + #[cfg(not(target_arch = "aarch64"))] + fn finalize_x86(&mut self) -> Result<(), OutOfRange> { for fixup in self.fixups.drain(..) { let origin = fixup.instruction_offset + fixup.instruction_length as usize; let target_absolute = self.labels[fixup.target_label.raw() as usize]; @@ -333,20 +373,82 @@ impl Assembler { let p = fixup.instruction_offset + fixup_offset as usize; if fixup_length == 1 { if offset > i8::MAX as isize || offset < i8::MIN as isize { - panic!("out of range jump"); + return Err(OutOfRange("jump")); } self.code[p] = offset as i8 as u8; } else if fixup_length == 4 { if offset > i32::MAX as isize || offset < i32::MIN as isize { - panic!("out of range jump"); + return Err(OutOfRange("jump")); } self.code[p..p + 4].copy_from_slice(&(offset as i32).to_le_bytes()); } else { unreachable!() } } + } + + /// Resolves AArch64 fixups: offset bitfield packed inside the 4-byte word, relative to instruction start. + #[cfg(target_arch = "aarch64")] + fn finalize_aarch64(&mut self) -> Result<(), OutOfRange> { + for fixup in self.fixups.drain(..) { + let target_absolute = self.labels[fixup.target_label.raw() as usize]; + if target_absolute == isize::MAX { + log::trace!("Undefined label found: {}", fixup.target_label); + continue; + } + + // Both ADR and branches are PC-relative to the start of the instruction. + let offset = target_absolute - fixup.instruction_offset as isize; + let p = fixup.instruction_offset; + let mut word = u32::from_le_bytes([self.code[p], self.code[p + 1], self.code[p + 2], self.code[p + 3]]); + + if fixup.kind.aarch64_is_adrp() { + // Page-relative: ADRP always uses 4 KiB pages, and the offset is between the pages + // of the *absolute* addresses, so `origin` matters here (unlike every other fixup). + let pc_page = self.origin.wrapping_add(fixup.instruction_offset as u64) >> 12; + let target_page = self.origin.wrapping_add(target_absolute as u64) >> 12; + let pages = (target_page as i64).wrapping_sub(pc_page as i64); + let limit = 1i64 << 20; + if pages >= limit || pages < -limit { + return Err(OutOfRange("ADRP")); + } + let imm = (pages as u32) & 0x001f_ffff; + let immlo = imm & 0b11; + let immhi = (imm >> 2) & 0x7_ffff; + word &= !((0b11 << 29) | (0x7_ffff << 5)); + word |= (immlo << 29) | (immhi << 5); + } else if fixup.kind.aarch64_is_adr() { + // 21-bit signed *byte* offset, split into immlo (bits 30:29) and immhi (bits 23:5). + let limit = 1isize << 20; + if offset >= limit || offset < -limit { + return Err(OutOfRange("ADR")); + } + let imm = (offset as u32) & 0x001f_ffff; + let immlo = imm & 0b11; + let immhi = (imm >> 2) & 0x7_ffff; + word &= !((0b11 << 29) | (0x7_ffff << 5)); + word |= (immlo << 29) | (immhi << 5); + } else { + let lsb = fixup.kind.aarch64_lsb(); + let width = fixup.kind.aarch64_width(); + + debug_assert_eq!(offset & 0b11, 0, "AArch64 branch target is not 4-byte aligned"); + let imm = offset >> 2; + + // Range-check against the signed field width. + let limit = 1isize << (width - 1); + if imm >= limit || imm < -limit { + return Err(OutOfRange("jump")); + } + + let mask = ((1u32 << width) - 1) << lsb; + word = (word & !mask) | (((imm as u32) << lsb) & mask); + } + + self.code[p..p + 4].copy_from_slice(&word.to_le_bytes()); + } - AssembledCode(self) + Ok(()) } pub fn is_empty(&self) -> bool { diff --git a/crates/polkavm-assembler/src/lib.rs b/crates/polkavm-assembler/src/lib.rs index 140d7280b..acfc4c2b2 100644 --- a/crates/polkavm-assembler/src/lib.rs +++ b/crates/polkavm-assembler/src/lib.rs @@ -5,6 +5,9 @@ pub mod amd64; +#[cfg(target_arch = "aarch64")] +pub mod aarch64; + #[cfg(feature = "alloc")] mod assembler; mod misc; @@ -13,5 +16,5 @@ mod misc; extern crate alloc; #[cfg(feature = "alloc")] -pub use crate::assembler::{Assembler, NonZero, ReservedAssembler, U0, U1, U2, U3, U4, U5, U6}; +pub use crate::assembler::{Assembler, NonZero, OutOfRange, ReservedAssembler, U0, U1, U2, U3, U4, U5, U6}; pub use crate::misc::{InstructionT, Label}; diff --git a/crates/polkavm-assembler/src/misc.rs b/crates/polkavm-assembler/src/misc.rs index b825006de..2ec1a11e9 100644 --- a/crates/polkavm-assembler/src/misc.rs +++ b/crates/polkavm-assembler/src/misc.rs @@ -73,12 +73,14 @@ impl From for InstructionOrBuffer { pub struct FixupKind(pub u32); impl FixupKind { + #[cfg(not(target_arch = "aarch64"))] #[cfg_attr(not(feature = "alloc"), allow(dead_code))] #[inline] pub const fn offset(self) -> u32 { (self.0 >> 24) & 0b11 } + #[cfg(not(target_arch = "aarch64"))] #[cfg_attr(not(feature = "alloc"), allow(dead_code))] #[inline] pub const fn length(self) -> u32 { @@ -101,6 +103,57 @@ impl FixupKind { let opcode = opcode[0] | (opcode[1] << 8) | (opcode[2] << 16); FixupKind((3 << 24) | (length << 28) | opcode) } + + /// AArch64 branch fixup: signed `width`-bit immediate at bit `lsb`, scaled by 4, relative to the + /// instruction start. Same `u32` storage as the x86 constructors; `finalize` interprets per-arch. + #[cfg(target_arch = "aarch64")] + #[inline] + pub const fn aarch64_branch(lsb: u32, width: u32) -> Self { + FixupKind(lsb | (width << 8)) + } + + /// `ADR` fixup: signed 21-bit byte offset split across `immlo` (30:29) / `immhi` (23:5); tagged by bit 16. + #[cfg(target_arch = "aarch64")] + #[inline] + pub const fn aarch64_adr() -> Self { + FixupKind(1 << 16) + } + + /// `ADRP` fixup: same immediate fields as `ADR`, but a signed 21-bit *4 KiB page* offset + /// (±4 GiB) computed from absolute addresses; tagged by bit 17. + #[cfg(target_arch = "aarch64")] + #[inline] + pub const fn aarch64_adrp() -> Self { + FixupKind(1 << 17) + } + + #[cfg(target_arch = "aarch64")] + #[cfg_attr(not(feature = "alloc"), allow(dead_code))] + #[inline] + pub const fn aarch64_is_adr(self) -> bool { + (self.0 >> 16) & 1 == 1 + } + + #[cfg(target_arch = "aarch64")] + #[cfg_attr(not(feature = "alloc"), allow(dead_code))] + #[inline] + pub const fn aarch64_is_adrp(self) -> bool { + (self.0 >> 17) & 1 == 1 + } + + #[cfg(target_arch = "aarch64")] + #[cfg_attr(not(feature = "alloc"), allow(dead_code))] + #[inline] + pub const fn aarch64_lsb(self) -> u32 { + self.0 & 0xff + } + + #[cfg(target_arch = "aarch64")] + #[cfg_attr(not(feature = "alloc"), allow(dead_code))] + #[inline] + pub const fn aarch64_width(self) -> u32 { + (self.0 >> 8) & 0xff + } } const MAXIMUM_INSTRUCTION_SIZE: usize = 16; diff --git a/crates/polkavm-common/src/lib.rs b/crates/polkavm-common/src/lib.rs index a01f9699b..e17796b46 100644 --- a/crates/polkavm-common/src/lib.rs +++ b/crates/polkavm-common/src/lib.rs @@ -31,7 +31,7 @@ pub mod varint; #[cfg(feature = "alloc")] pub mod writer; -#[cfg(target_arch = "x86_64")] +#[cfg(any(target_arch = "x86_64", target_arch = "aarch64"))] pub mod zygote; #[cfg(feature = "regmap")] diff --git a/crates/polkavm-common/src/regmap.rs b/crates/polkavm-common/src/regmap.rs index e931de70c..895aca54e 100644 --- a/crates/polkavm-common/src/regmap.rs +++ b/crates/polkavm-common/src/regmap.rs @@ -1,7 +1,16 @@ use crate::program::Reg; + +#[cfg(target_arch = "x86_64")] pub use polkavm_assembler::amd64::RegIndex as NativeReg; +#[cfg(target_arch = "x86_64")] use polkavm_assembler::amd64::RegIndex::*; +#[cfg(target_arch = "aarch64")] +pub use polkavm_assembler::aarch64::Reg as NativeReg; +#[cfg(target_arch = "aarch64")] +use polkavm_assembler::aarch64::Reg::*; + +#[cfg(target_arch = "x86_64")] #[inline] pub const fn to_native_reg(reg: Reg) -> NativeReg { // NOTE: This is sorted roughly in the order of which registers are more commonly used. @@ -24,11 +33,49 @@ pub const fn to_native_reg(reg: Reg) -> NativeReg { } /// A temporary register which can be freely used. +#[cfg(target_arch = "x86_64")] pub const TMP_REG: NativeReg = rcx; /// A temporary register which must be saved/restored. +#[cfg(target_arch = "x86_64")] pub const AUX_TMP_REG: NativeReg = r13; +#[cfg(target_arch = "aarch64")] +#[inline] +pub const fn to_native_reg(reg: Reg) -> NativeReg { + // The 13 guest registers are kept live in native registers during guest execution (loaded at + // sysenter, stored back at sysreturn). They are placed in the caller-saved bank (x0..x12): this + // lets the sandbox entry inline assembly clobber them via `clobber_abi("C")` without having to + // touch the callee-saved registers x19..x28, which Rust/LLVM forbids from appearing as inline-asm + // operands (x19 in particular is reserved by LLVM). We deliberately avoid x18 (the platform + // register, reserved on Darwin), x29 (frame pointer), x30 (link register) and the stack pointer; + // x15..x17 are left free as codegen scratch. + match reg { + Reg::A0 => X0, + Reg::A1 => X1, + Reg::A2 => X2, + Reg::A3 => X3, + Reg::A4 => X4, + Reg::A5 => X5, + Reg::S0 => X6, + Reg::S1 => X7, + Reg::SP => X8, + Reg::RA => X9, + Reg::T0 => X10, + Reg::T1 => X11, + Reg::T2 => X12, + } +} + +/// A temporary register which can be freely used (analogous to `rcx` on x86-64). +#[cfg(target_arch = "aarch64")] +pub const TMP_REG: NativeReg = X13; + +/// A temporary register which must be saved/restored; on the generic sandbox it holds the base +/// address of the guest's linear memory (analogous to `r15` on x86-64). +#[cfg(target_arch = "aarch64")] +pub const AUX_TMP_REG: NativeReg = X14; + #[inline] pub const fn to_guest_reg(reg: NativeReg) -> Option { let mut index = 0; diff --git a/crates/polkavm-common/src/zygote.rs b/crates/polkavm-common/src/zygote.rs index f419e6114..3bb182128 100644 --- a/crates/polkavm-common/src/zygote.rs +++ b/crates/polkavm-common/src/zygote.rs @@ -124,6 +124,10 @@ pub const VM_SHARED_MEMORY_SIZE: u64 = u32::MAX as u64; /// /// This does *not* affect the VM ABI and can be changed at will, /// but should be high enough that it's never hit. +// AArch64 needs more headroom (fixed 4-byte insns, immediate expansion, gas stub on first insn). +#[cfg(target_arch = "aarch64")] +pub const VM_COMPILER_MAXIMUM_INSTRUCTION_LENGTH: u32 = 112; +#[cfg(not(target_arch = "aarch64"))] pub const VM_COMPILER_MAXIMUM_INSTRUCTION_LENGTH: u32 = 69; /// The maximum number of bytes the jump table can be. @@ -136,6 +140,10 @@ pub const VM_SANDBOX_MAXIMUM_JUMP_TABLE_VIRTUAL_SIZE: u64 = 0x100000000 * core:: // TODO: Make this smaller. /// The maximum number of bytes the native code can be. +// Must be >= VM_MAXIMUM_CODE_SIZE * VM_COMPILER_MAXIMUM_INSTRUCTION_LENGTH (see static assert below). +#[cfg(target_arch = "aarch64")] +pub const VM_SANDBOX_MAXIMUM_NATIVE_CODE_SIZE: u32 = 3584 * 1024 * 1024; // 32 MiB * 112 +#[cfg(not(target_arch = "aarch64"))] pub const VM_SANDBOX_MAXIMUM_NATIVE_CODE_SIZE: u32 = 2303 * 1024 * 1024 - 1; #[repr(C)] diff --git a/crates/polkavm/Cargo.toml b/crates/polkavm/Cargo.toml index ccbc91ecd..340e54181 100644 --- a/crates/polkavm/Cargo.toml +++ b/crates/polkavm/Cargo.toml @@ -20,7 +20,7 @@ schnellru = { workspace = true, optional = true } [target.'cfg(all(not(miri), target_arch = "x86_64", target_os = "linux"))'.dependencies] polkavm-linux-raw = { workspace = true, features = ["std"] } -[target.'cfg(all(not(miri), target_arch = "x86_64", any(target_os = "macos", target_os = "freebsd")))'.dependencies] +[target.'cfg(all(not(miri), any(all(target_arch = "x86_64", any(target_os = "macos", target_os = "freebsd")), all(target_arch = "aarch64", any(target_os = "macos", target_os = "linux")))))'.dependencies] libc = { workspace = true } [dev-dependencies] @@ -48,6 +48,10 @@ module-cache = ["dep:schnellru", "polkavm-common/blake3"] # This sandbox is EXPERIMENTAL and is not meant for production use. generic-sandbox = [] +# Hardware-virtualization sandbox (aarch64-only): runs the guest in a vCPU with its +# own stage-1 MMU, giving 4K guest pages on any host. EXPERIMENTAL; currently a scaffold. +hypervisor-sandbox = [] + # Internal feature for testing. DO NOT USE. export-internals-for-testing = [] diff --git a/crates/polkavm/src/api.rs b/crates/polkavm/src/api.rs index 07764f164..e7cdad42d 100644 --- a/crates/polkavm/src/api.rs +++ b/crates/polkavm/src/api.rs @@ -30,10 +30,12 @@ if_compiler_is_supported! { use crate::sandbox::{Sandbox, SandboxInstance}; use crate::compiler::{CompiledModule, CompilerCache}; - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] use crate::sandbox::linux::Sandbox as SandboxLinux; #[cfg(feature = "generic-sandbox")] use crate::sandbox::generic::Sandbox as SandboxGeneric; + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + use crate::sandbox::hypervisor::Sandbox as SandboxHypervisor; pub(crate) struct EngineState { pub(crate) sandboxing_enabled: bool, @@ -57,7 +59,7 @@ trait IntoResult { } if_compiler_is_supported! { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] impl IntoResult for Result { fn into_result(self, message: &str) -> Result { self.map_err(|error| Error::from(error).context(message)) @@ -73,6 +75,13 @@ if_compiler_is_supported! { self.map_err(|error| Error::from(error).context(message)) } } + + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + impl IntoResult for Result { + fn into_result(self, message: &str) -> Result { + self.map_err(|error| Error::from(error).context(message)) + } + } } impl IntoResult for T { @@ -156,6 +165,8 @@ impl Engine { if selected_backend == BackendKind::Compiler { let default_sandbox = if SandboxKind::Linux.is_supported() { SandboxKind::Linux + } else if SandboxKind::Hypervisor.is_supported() { + SandboxKind::Hypervisor } else { SandboxKind::Generic }; @@ -167,7 +178,7 @@ impl Engine { bail!("the '{selected_sandbox}' backend is not supported on this platform") } - if selected_sandbox == SandboxKind::Generic && !config.allow_experimental { + if matches!(selected_sandbox, SandboxKind::Generic | SandboxKind::Hypervisor) && !config.allow_experimental { bail!("cannot use the '{selected_sandbox}' sandbox: this sandbox is not production ready and may be insecure; you can enabled `set_allow_experimental`/`POLKAVM_ALLOW_EXPERIMENTAL` to be able to use it anyway"); } @@ -239,10 +250,12 @@ impl Engine { if_compiler_is_supported! { { pub(crate) enum CompiledModuleKind { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] Linux(CompiledModule), #[cfg(feature = "generic-sandbox")] Generic(CompiledModule), + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + Hypervisor(CompiledModule), Unavailable, } } else { @@ -392,6 +405,8 @@ impl Module { } if_compiler_is_supported! { + // Used by the generic sandbox's page-fault path. + #[cfg_attr(not(feature = "generic-sandbox"), allow(dead_code))] pub(crate) fn address_to_page(&self, address: u32) -> u32 { address >> self.state().page_shift } @@ -590,7 +605,7 @@ impl Module { if let Some(selected_sandbox) = engine.selected_sandbox { match selected_sandbox { SandboxKind::Linux => { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] match blob.isa() { InstructionSetKind::ReviveV1 => compile_module!(SandboxLinux, B64, build_static_dispatch_table_revive_v1, COMPILER_VISITOR_LINUX, Linux), InstructionSetKind::JamV1 => compile_module!(SandboxLinux, B64, build_static_dispatch_table_jam_v1, COMPILER_VISITOR_LINUX, Linux), @@ -598,7 +613,7 @@ impl Module { InstructionSetKind::Latest64 => compile_module!(SandboxLinux, B64, build_static_dispatch_table_latest64, COMPILER_VISITOR_LINUX, Linux), } - #[cfg(not(target_os = "linux"))] + #[cfg(not(all(target_os = "linux", target_arch = "x86_64")))] { log::debug!("Selecetd sandbox unavailable: 'linux'"); None @@ -619,6 +634,21 @@ impl Module { None } }, + SandboxKind::Hypervisor => { + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + match blob.isa() { + InstructionSetKind::ReviveV1 => compile_module!(SandboxHypervisor, B64, build_static_dispatch_table_revive_v1, COMPILER_VISITOR_HYPERVISOR, Hypervisor), + InstructionSetKind::JamV1 => compile_module!(SandboxHypervisor, B64, build_static_dispatch_table_jam_v1, COMPILER_VISITOR_HYPERVISOR, Hypervisor), + InstructionSetKind::Latest32 => compile_module!(SandboxHypervisor, B32, build_static_dispatch_table_latest32, COMPILER_VISITOR_HYPERVISOR, Hypervisor), + InstructionSetKind::Latest64 => compile_module!(SandboxHypervisor, B64, build_static_dispatch_table_latest64, COMPILER_VISITOR_HYPERVISOR, Hypervisor), + } + + #[cfg(not(all(feature = "hypervisor-sandbox", target_arch = "aarch64")))] + { + log::debug!("Selected sandbox unavailable: 'hypervisor'"); + None + } + }, } } else { None @@ -749,7 +779,7 @@ impl Module { let backend = if_compiler_is_supported! { {{ match compiled_module { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] CompiledModuleKind::Linux(..) => { let outer_instance = match outer_instance { Some(outer_instance) => { @@ -781,6 +811,22 @@ impl Module { let compiled_instance = SandboxInstance::::spawn_and_load_module(Arc::clone(engine_state), self, outer_instance)?; Some(InstanceBackend::CompiledGeneric(compiled_instance)) }, + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + CompiledModuleKind::Hypervisor(..) => { + let outer_instance = match outer_instance { + Some(outer_instance) => { + let outer_module = &outer_instance.module; + #[allow(clippy::match_wildcard_for_single_variants)] + match outer_instance.backend { + InstanceBackend::CompiledHypervisor(ref outer_instance) if outer_module.engine_state_pointer() == self.engine_state_pointer() => Some(outer_instance), + _ => return Err(Error::from_static_str("failed to instantiate module: received incompatible outer instance")), + } + }, + None => None, + }; + let compiled_instance = SandboxInstance::::spawn_and_load_module(Arc::clone(engine_state), self, outer_instance)?; + Some(InstanceBackend::CompiledHypervisor(compiled_instance)) + }, CompiledModuleKind::Unavailable => None } }} else { @@ -862,10 +908,12 @@ impl Module { if_compiler_is_supported! { { match self.state().compiled_module { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] CompiledModuleKind::Linux(ref module) => Some(module.machine_code()), #[cfg(feature = "generic-sandbox")] CompiledModuleKind::Generic(ref module) => Some(module.machine_code()), + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + CompiledModuleKind::Hypervisor(ref module) => Some(module.machine_code()), CompiledModuleKind::Unavailable => None, } } else { @@ -882,10 +930,12 @@ impl Module { if_compiler_is_supported! { { match self.state().compiled_module { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] CompiledModuleKind::Linux(..) => Some(polkavm_common::zygote::VM_ADDR_NATIVE_CODE), #[cfg(feature = "generic-sandbox")] CompiledModuleKind::Generic(..) => None, + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + CompiledModuleKind::Hypervisor(..) => None, CompiledModuleKind::Unavailable => None, } } else { @@ -911,10 +961,12 @@ impl Module { if_compiler_is_supported! { { match self.state().compiled_module { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] CompiledModuleKind::Linux(ref module) => Some(module.program_counter_to_machine_code_offset()), #[cfg(feature = "generic-sandbox")] CompiledModuleKind::Generic(ref module) => Some(module.program_counter_to_machine_code_offset()), + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + CompiledModuleKind::Hypervisor(ref module) => Some(module.program_counter_to_machine_code_offset()), CompiledModuleKind::Unavailable => None, } } else { @@ -1032,10 +1084,12 @@ impl Module { if_compiler_is_supported! { { enum InstanceBackend { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] CompiledLinux(SandboxInstance), #[cfg(feature = "generic-sandbox")] CompiledGeneric(SandboxInstance), + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + CompiledHypervisor(SandboxInstance), Interpreted(InterpretedInstance), } } else { @@ -1130,7 +1184,7 @@ if_compiler_is_supported! { macro_rules! access_backend { ($itself:expr, |$backend:ident| $e:expr) => { match $itself { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] InstanceBackend::CompiledLinux(ref $backend) => { let $backend = $backend.sandbox(); $e @@ -1140,13 +1194,18 @@ if_compiler_is_supported! { let $backend = $backend.sandbox(); $e }, + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + InstanceBackend::CompiledHypervisor(ref $backend) => { + let $backend = $backend.sandbox(); + $e + }, InstanceBackend::Interpreted(ref $backend) => $e, } }; ($itself:expr, |mut $backend:ident| $e:expr) => { match $itself { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] InstanceBackend::CompiledLinux(ref mut $backend) => { let $backend = $backend.sandbox_mut(); $e @@ -1156,6 +1215,11 @@ if_compiler_is_supported! { let $backend = $backend.sandbox_mut(); $e }, + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + InstanceBackend::CompiledHypervisor(ref mut $backend) => { + let $backend = $backend.sandbox_mut(); + $e + }, InstanceBackend::Interpreted(ref mut $backend) => $e, } }; diff --git a/crates/polkavm/src/compiler.rs b/crates/polkavm/src/compiler.rs index e281f6386..2312d7bf4 100644 --- a/crates/polkavm/src/compiler.rs +++ b/crates/polkavm/src/compiler.rs @@ -26,6 +26,18 @@ pub use crate::compiler::amd64::{extract_gas_cost, on_page_fault, on_signal_trap #[cfg(all(target_arch = "x86_64", feature = "generic-sandbox"))] pub(crate) use crate::compiler::amd64::{are_we_executing_memset, indirect_memory_operand, MemsetKind}; +#[cfg(target_arch = "aarch64")] +mod aarch64; + +#[cfg(target_arch = "aarch64")] +pub use crate::compiler::aarch64::{extract_gas_cost, step_prelude_length}; + +#[cfg(all(target_arch = "aarch64", feature = "generic-sandbox"))] +pub(crate) use crate::compiler::aarch64::{are_we_executing_memset, MemsetKind}; + +#[cfg(all(target_arch = "aarch64", feature = "hypervisor-sandbox"))] +pub(crate) use crate::compiler::aarch64::GAS_METERING_TRAP_OFFSET; + /// The address to which to jump to for invalid dynamic jumps. /// /// This needs to be at least 0x800000000000 on modern CPUs, but ideally should have @@ -106,6 +118,8 @@ where rem32s_label: Label, rem64u_label: Label, rem64s_label: Label, + // Used by the x86 backend; the AArch64 backend traps invalid targets via `trap_label` instead. + #[allow(dead_code)] invalid_jump_label: Label, instruction_set: InstructionSetKind, last_basic_block_start: u32, @@ -171,7 +185,14 @@ where S: Sandbox, { let native_page_size = crate::sandbox::get_native_page_size(); - if native_page_size > config.page_size as usize || config.page_size as usize % native_page_size != 0 { + // macOS (dev only) also allows a guest page smaller than the 16K Apple-Silicon native page, as + // long as one divides the other; bounds checks are then host-page-coarse (see the sandbox). + let compatible = if cfg!(target_os = "macos") { + config.page_size as usize % native_page_size == 0 || native_page_size % config.page_size as usize == 0 + } else { + native_page_size <= config.page_size as usize && config.page_size as usize % native_page_size == 0 + }; + if !compatible { return Err(format!( "configured page size of {} is incompatible with the native page size of {}", config.page_size, native_page_size @@ -419,18 +440,28 @@ where match S::KIND { SandboxKind::Linux => {} - SandboxKind::Generic => { + SandboxKind::Generic | SandboxKind::Hypervisor => { let native_page_size = crate::sandbox::get_native_page_size(); let padded_length = polkavm_common::utils::align_to_next_page_usize(native_page_size, self.asm.len()).unwrap(); self.asm.resize(padded_length, ArchVisitor::::PADDING_BYTE); + // The AArch64 codegen reaches the table with a bare `adrp`, which is 4K-page granular. + debug_assert_eq!(padded_length % 4096, 0); + debug_assert_eq!(native_code_origin % 4096, 0); self.asm.define_label(self.jump_table_label); } } let module = { + // An out-of-range fixup means the program was too big for the backend's branch + // encodings: a property of the input, not a bug, so report it as a failed compilation. + let code = self + .asm + .try_finalize() + .map_err(|error| Error::from(alloc::format!("failed to compile the program: {error}")))?; + let init = SandboxInit { guest_init: self.init, - code: &self.asm.finalize(), + code: &code, jump_table: native_jump_table, sysenter_address, sysreturn_address, @@ -591,6 +622,7 @@ where } } + #[allow(dead_code)] // Used by the x86 backend; the AArch64 backend defines labels via the assembler directly. fn define_label(&mut self, label: Label) { log::trace!("Label: {} -> {:08x}", label, self.asm.current_address()); self.asm.define_label(label); @@ -1601,10 +1633,13 @@ where // Basic block offsets. gas_metering_stub_offsets: Vec, cache: CompilerCache, - step_tracing: bool, + pub(crate) step_tracing: bool, + // Read by the x86 backend / step tracing; unused on the generic AArch64 build. + #[allow(dead_code)] pub(crate) bitness: Bitness, - + #[allow(dead_code)] pub(crate) memset_trampoline_start: u64, + #[allow(dead_code)] pub(crate) memset_trampoline_end: u64, } @@ -1695,6 +1730,44 @@ where .map(|native_offset| self.native_code_origin + u64::from(native_offset)) } + /// Native address to resume at after a page-fault. AArch64 skips the block prologue so gas + /// isn't re-charged; on x86 the faulting instruction is self-contained. + #[cfg(any(feature = "generic-sandbox", feature = "hypervisor-sandbox"))] + #[allow(unused_variables, clippy::unused_self)] + pub(crate) fn resume_native_address_for_pagefault( + &self, + program_counter: ProgramCounter, + machine_code_address: u64, + gas_metering: Option, + ) -> u64 { + #[cfg(not(target_arch = "aarch64"))] + { + machine_code_address + } + #[cfg(target_arch = "aarch64")] + { + let Some(base) = self.lookup_native_code_address(program_counter) else { + return machine_code_address; + }; + // Resume at the faulting instruction's body (past the step prelude): its address/value are + // recomputed from the (possibly host-changed) guest registers, and the step prelude must NOT + // re-fire (the interpreter re-executes the instruction without a new step on resume). + let mut resume = base; + if self.step_tracing { + resume += step_prelude_length::() as u64; + } + // If the resume point is a block's gas-metering stub (i.e. this is a block-first + // instruction), skip past it so the stub isn't re-executed and gas isn't re-charged. + if let Some(kind) = gas_metering { + let offset = (resume - self.native_code_origin) as u32; + if self.gas_metering_stub_offsets.binary_search(&offset).is_ok() { + resume += crate::compiler::aarch64::gas_metering_stub_length(kind) as u64; + } + } + resume + } + } + pub fn program_counter_by_native_code_offset(&self, offset: u64, strict: bool) -> Option { let index = match self .program_counter_to_machine_code_offset_list diff --git a/crates/polkavm/src/compiler/aarch64.rs b/crates/polkavm/src/compiler/aarch64.rs new file mode 100644 index 000000000..9cde84e33 --- /dev/null +++ b/crates/polkavm/src/compiler/aarch64.rs @@ -0,0 +1,1408 @@ +//! AArch64 (ARMv8.2-A) code generation backend; ARM counterpart to [`crate::compiler::amd64`]. + +use polkavm_assembler::aarch64 as a64; +use polkavm_assembler::Label; + +use polkavm_common::program::{RawReg, Reg}; +use polkavm_common::regmap::{to_native_reg, NativeReg, AUX_TMP_REG, TMP_REG}; +use polkavm_common::utils::GasVisitorT; + +use crate::compiler::{ArchVisitor, Bitness, BitnessT, SandboxKind}; +use crate::config::GasMeteringKind; +use crate::sandbox::Sandbox; + +/// Converts a guest register reference into its native home register. +#[inline] +fn conv_reg(reg: RawReg) -> NativeReg { + to_native_reg(reg.get()) +} + +/// How a guest memory access is addressed; see `guest_access`. +enum GuestAccess { + /// `[x14, #imm]`, where `imm` is already scaled by the access size. + Offset(u32), + /// `[reg]`, with the full address materialized in `reg`. + Rebased(NativeReg), + /// `[x14, wreg, uxtw]`. + Index(NativeReg), +} + +/// Splits a signed constant into the add/sub-immediate encoding: `(imm12, lsl #12, is_sub)`. +fn add_sub_imm(value: i64) -> Option<(u32, bool, bool)> { + let sub = value < 0; + let mag = value.unsigned_abs(); + if mag < 0x1000 { + Some((mag as u32, false, sub)) + } else if mag & 0xfff == 0 && (mag >> 12) < 0x1000 { + Some(((mag >> 12) as u32, true, sub)) + } else { + None + } +} + +/// Guest linear-memory base (generic sandbox); the vmctx is at a fixed negative offset from it. +const MEMORY_BASE_REG: NativeReg = AUX_TMP_REG; // x14 + +// Codegen scratch (not guest regs x0..x12, TMP_REG x13 or MEMORY_BASE_REG x14). +const SCRATCH0: NativeReg = NativeReg::X16; +const SCRATCH1: NativeReg = NativeReg::X17; +const SCRATCH2: NativeReg = NativeReg::X15; + +/// Hardware stack pointer (encoding 31, shared with `XZR`); only for load/store and add/sub-immediate. +const SP: NativeReg = NativeReg::XZR; + +// Byte offsets within the gas-metering stub (see emit_gas_metering_stub). +const GAS_COST_OFFSET: usize = 8; // the embedded 32-bit cost literal +pub(crate) const GAS_METERING_TRAP_OFFSET: u64 = 36; // the `udf` that fires on gas underflow + +impl<'r, 'a, S, B, G> ArchVisitor<'r, 'a, S, B, G> +where + S: Sandbox, + B: BitnessT, + G: GasVisitorT, +{ + // Code/jump-table padding (never executed); `0x00` decodes as `udf #0` so any fall-through traps. + pub const PADDING_BYTE: u8 = 0x00; + + #[inline(always)] + fn push(&mut self, inst: impl polkavm_assembler::InstructionT) { + self.0.asm.push(inst); + } + + #[allow(clippy::unused_self)] + fn reg_size(&self) -> a64::RegSize { + match B::BITNESS { + Bitness::B32 => a64::RegSize::W, + Bitness::B64 => a64::RegSize::X, + } + } + + // ---- Core code-generation helpers ---- + + /// Materializes a 64-bit constant via `movz`/`movn` + `movk` (picks the fewer-`movk` base). + fn mov_imm(&mut self, rd: NativeReg, value: u64) { + use a64::RegSize::X; + let halves = [value as u16, (value >> 16) as u16, (value >> 32) as u16, (value >> 48) as u16]; + let ones = halves.iter().filter(|&&h| h == 0xffff).count(); + let zeros = halves.iter().filter(|&&h| h == 0).count(); + + if ones > zeros { + // `movn rd, #imm, lsl s` sets that halfword and fills the rest with ones. + let first = halves.iter().position(|&h| h != 0xffff).unwrap_or(0); + self.push(a64::movn(X, rd, !halves[first], first as u32)); + for (i, &half) in halves.iter().enumerate() { + if i != first && half != 0xffff { + self.push(a64::movk(X, rd, half, i as u32)); + } + } + } else { + let first = halves.iter().position(|&h| h != 0).unwrap_or(0); + self.push(a64::movz(X, rd, halves[first], first as u32)); + for (i, &half) in halves.iter().enumerate() { + if i > first && half != 0 { + self.push(a64::movk(X, rd, half, i as u32)); + } + } + } + } + + /// `rd = rn + value` (signed); single add/sub immediate (incl. `lsl #12`) when it fits, else materialize. + fn add_const(&mut self, rd: NativeReg, rn: NativeReg, value: u64) { + use a64::RegSize::X; + let signed = value as i64; + if signed == 0 { + if !rd.equals(rn) { + self.push(a64::mov_reg(X, rd, rn)); + } + return; + } + + if let Some((imm, shift12, sub)) = add_sub_imm(signed) { + if sub { + self.push(a64::sub_imm(X, rd, rn, imm, shift12)); + } else { + self.push(a64::add_imm(X, rd, rn, imm, shift12)); + } + } else { + self.mov_imm(SCRATCH1, value); + self.push(a64::add_reg(X, rd, rn, SCRATCH1)); + } + } + + /// `rd = rn + value` at `size`, preferring the add/sub-immediate form. Adding a negative value + /// is subtracting its magnitude, which is the same modulo the register width. + fn add_const_sized(&mut self, size: a64::RegSize, rd: NativeReg, rn: NativeReg, value: i64) { + if value == 0 { + if !rd.equals(rn) { + self.push(a64::mov_reg(size, rd, rn)); + } + return; + } + if let Some((imm, shift12, sub)) = add_sub_imm(value) { + if sub { + self.push(a64::sub_imm(size, rd, rn, imm, shift12)); + } else { + self.push(a64::add_imm(size, rd, rn, imm, shift12)); + } + } else { + self.mov_imm(SCRATCH0, value as u64); + self.push(a64::add_reg(size, rd, rn, SCRATCH0)); + } + } + + /// `cmp rn, #value` when the immediate is encodable, else via a scratch register. + fn cmp_const(&mut self, size: a64::RegSize, rn: NativeReg, value: i64) { + // Only the non-negative form is encodable as `subs`; a negative would need `cmn`. + match add_sub_imm(value) { + Some((imm, shift12, false)) => self.push(a64::cmp_imm(size, rn, imm, shift12)), + _ => { + self.mov_imm(SCRATCH0, value as u64); + self.push(a64::cmp_reg(size, rn, SCRATCH0)); + } + } + } + + /// An immediate operand as a signed 64-bit value; RISC-V immediates are sign-extended, and in + /// 32-bit mode the consumers only look at the low half either way. + #[allow(clippy::unused_self)] + fn imm_operand(&self, value: u32) -> i64 { + i64::from(value as i32) + } + + /// Bitmask-immediate encoding of an immediate operand as the guest sees it (sign-extended in + /// 64-bit mode, zero-extended in 32-bit mode), if the value has one. + fn logical_imm(&self, value: u32) -> Option { + let value = match B::BITNESS { + Bitness::B32 => u64::from(value), + Bitness::B64 => i64::from(value as i32) as u64, + }; + a64::logical_imm(value, self.reg_size()) + } + + /// Address of a VM-context field into `dst`. Both the generic and hypervisor sandboxes + /// place the vmctx one page below the guest memory base, so they share this codegen. + fn vmctx_addr(&mut self, dst: NativeReg, field_offset: usize) { + debug_assert!(matches!(S::KIND, SandboxKind::Generic | SandboxKind::Hypervisor)); + let off = (crate::sandbox::GUEST_MEMORY_TO_VMCTX_OFFSET as i64 + field_offset as i64) as u64; + self.add_const(dst, MEMORY_BASE_REG, off); + } + + fn ld_vmctx(&mut self, dst: NativeReg, field_offset: usize) { + self.vmctx_addr(SCRATCH0, field_offset); + self.push(a64::ldr_imm(a64::Size::U64, dst, SCRATCH0, 0)); + } + + fn st_vmctx(&mut self, src: NativeReg, field_offset: usize) { + self.vmctx_addr(SCRATCH0, field_offset); + self.push(a64::str_imm(a64::Size::U64, src, SCRATCH0, 0)); + } + + fn st_vmctx_imm(&mut self, value: u64, field_offset: usize) { + if value == 0 { + self.vmctx_addr(SCRATCH0, field_offset); + self.push(a64::str_imm(a64::Size::U64, NativeReg::XZR, SCRATCH0, 0)); + } else { + self.mov_imm(SCRATCH2, value); + self.vmctx_addr(SCRATCH0, field_offset); + self.push(a64::str_imm(a64::Size::U64, SCRATCH2, SCRATCH0, 0)); + } + } + + fn st_vmctx_imm32(&mut self, value: u32, field_offset: usize) { + if value == 0 { + self.vmctx_addr(SCRATCH0, field_offset); + self.push(a64::str_imm(a64::Size::U32, NativeReg::XZR, SCRATCH0, 0)); + } else { + self.mov_imm(SCRATCH2, u64::from(value)); + self.vmctx_addr(SCRATCH0, field_offset); + self.push(a64::str_imm(a64::Size::U32, SCRATCH2, SCRATCH0, 0)); + } + } + + fn save_registers_to_vmctx(&mut self) { + self.vmctx_addr(SCRATCH0, S::offset_table().regs); + for (nth, reg) in Reg::ALL.iter().enumerate() { + self.push(a64::str_imm(a64::Size::U64, conv_reg((*reg).into()), SCRATCH0, nth as u32)); + } + } + + fn restore_registers_from_vmctx(&mut self) { + self.vmctx_addr(SCRATCH0, S::offset_table().regs); + for (nth, reg) in Reg::ALL.iter().enumerate() { + self.push(a64::ldr_imm(a64::Size::U64, conv_reg((*reg).into()), SCRATCH0, nth as u32)); + } + } + + /// Stores the link register (trampoline return address) into the VM context for host resumption. + fn save_return_address_to_vmctx(&mut self) { + self.st_vmctx(NativeReg::X30, S::offset_table().next_native_program_counter); + } + + /// Sign-extends a 32-bit result in `d` to 64 bits (RV64 word-op semantics); no-op in 32-bit mode. + fn finish32(&mut self, d: RawReg) { + if matches!(B::BITNESS, Bitness::B64) { + let d = conv_reg(d); + self.push(a64::sbfm(a64::RegSize::X, d, d, 0, 31)); + } + } + + /// Immediate into a scratch register, sign-extended (RISC-V immediates are signed; 32-bit consumers read the low half). + fn imm_reg(&mut self, value: u32) -> NativeReg { + self.mov_imm(SCRATCH0, i64::from(value as i32) as u64); + SCRATCH0 + } + + /// Addressing for a guest access at `(base + offset) as u32`: a scaled immediate off the memory + /// base, or a `uxtw` index — whose 32-bit extension *is* the address wrap, so no truncation. + fn guest_access(&mut self, base: Option, offset: u32, size: a64::Size) -> GuestAccess { + use a64::RegSize::W; + + let Some(base) = base else { + // Constant address: `[x14, #offset]` when the offset fits the scaled 12-bit field. + let scale = 1u32 << size.raw(); + if offset % scale == 0 && offset / scale < 0x1000 { + return GuestAccess::Offset(offset / scale); + } + self.add_const(SCRATCH0, MEMORY_BASE_REG, u64::from(offset)); + return GuestAccess::Rebased(SCRATCH0); + }; + + let base = conv_reg(base); + if offset == 0 { + // Only the low 32 bits of the index are used, so a dirty upper half is harmless. + return GuestAccess::Index(base); + } + if let Some((imm, shift12, sub)) = add_sub_imm(i64::from(offset as i32)) { + if sub { + self.push(a64::sub_imm(W, SCRATCH0, base, imm, shift12)); + } else { + self.push(a64::add_imm(W, SCRATCH0, base, imm, shift12)); + } + } else { + self.mov_imm(SCRATCH0, u64::from(offset)); + self.push(a64::add_reg(W, SCRATCH0, base, SCRATCH0)); + } + GuestAccess::Index(SCRATCH0) + } + + fn emit_guest_load(&mut self, dst: RawReg, base: Option, offset: u32, size: a64::Size, signed: bool) { + let access = self.guest_access(base, offset, size); + let d = conv_reg(dst); + // A signed word load only widens to 64 bits; in 32-bit mode a word load already fills the reg. + let plain = !signed || (matches!(size, a64::Size::U32) && matches!(B::BITNESS, Bitness::B32)); + match access { + GuestAccess::Offset(imm) => { + let rn = MEMORY_BASE_REG; + if plain { + self.push(a64::ldr_imm(size, d, rn, imm)); + } else { + match B::BITNESS { + Bitness::B32 => self.push(a64::ldrs32_imm(size, d, rn, imm)), + Bitness::B64 => self.push(a64::ldrs64_imm(size, d, rn, imm)), + } + } + } + GuestAccess::Rebased(rn) => { + if plain { + self.push(a64::ldr_imm(size, d, rn, 0)); + } else { + match B::BITNESS { + Bitness::B32 => self.push(a64::ldrs32_imm(size, d, rn, 0)), + Bitness::B64 => self.push(a64::ldrs64_imm(size, d, rn, 0)), + } + } + } + GuestAccess::Index(index) => { + let (rn, rm) = (MEMORY_BASE_REG, index); + if plain { + self.push(a64::ldr_reg(size, d, rn, rm, a64::Index::Uxtw)); + } else { + match B::BITNESS { + Bitness::B32 => self.push(a64::ldrs32_reg(size, d, rn, rm, a64::Index::Uxtw)), + Bitness::B64 => self.push(a64::ldrs64_reg(size, d, rn, rm, a64::Index::Uxtw)), + } + } + } + } + } + + fn emit_guest_store(&mut self, src: NativeReg, base: Option, offset: u32, size: a64::Size) { + match self.guest_access(base, offset, size) { + GuestAccess::Offset(imm) => self.push(a64::str_imm(size, src, MEMORY_BASE_REG, imm)), + GuestAccess::Rebased(rn) => self.push(a64::str_imm(size, src, rn, 0)), + GuestAccess::Index(index) => self.push(a64::str_reg(size, src, MEMORY_BASE_REG, index, a64::Index::Uxtw)), + } + } + + fn emit_guest_store_reg(&mut self, src: RawReg, base: Option, offset: u32, size: a64::Size) { + self.emit_guest_store(conv_reg(src), base, offset, size); + } + + fn emit_guest_store_imm(&mut self, value: u64, base: Option, offset: u32, size: a64::Size) { + // SCRATCH2 holds the value across the address computation (which only uses SCRATCH0/1). + let src = if value == 0 { + NativeReg::XZR + } else { + self.mov_imm(SCRATCH2, value); + SCRATCH2 + }; + self.emit_guest_store(src, base, offset, size); + } + + /// `rd = rn << n` (logical shift left by a constant), via the UBFM alias. + fn shl_imm(&mut self, size: a64::RegSize, rd: NativeReg, rn: NativeReg, n: u32) { + let bits = match size { + a64::RegSize::W => 32, + a64::RegSize::X => 64, + }; + self.push(a64::ubfm(size, rd, rn, (bits - n) % bits, bits - 1 - n)); + } + + /// Branch to `jump_table[(base + offset) as u32]` (opt. loading a return addr first). An invalid entry + /// faults at the target with PC lost, so we stash this site in next_native_program_counter for the handler. + fn jump_indirect_impl(&mut self, load_imm: Option<(RawReg, u32)>, base: RawReg, offset: u32) { + use a64::RegSize::{W, X}; + let here = self.0.asm.create_label(); + self.push(a64::adr(SCRATCH2, here)); // SCRATCH2: st_vmctx's addressing never clobbers it + self.st_vmctx(SCRATCH2, S::offset_table().next_native_program_counter); + + // index = (base + offset) as u32 + let b = conv_reg(base); + if offset == 0 { + self.push(a64::mov_reg(W, SCRATCH2, b)); + } else if offset < 0x1000 { + self.push(a64::add_imm(W, SCRATCH2, b, offset, false)); + } else { + self.mov_imm(SCRATCH2, u64::from(offset)); + self.push(a64::add_reg(W, SCRATCH2, b, SCRATCH2)); + } + self.shl_imm(X, SCRATCH2, SCRATCH2, 3); // index * 8 + + // `adrp` (±4 GiB), not `adr` (±1 MiB): the table sits past the end of the code, which can be + // far away. It is native-page-aligned (see `finish_compilation`), so the page base it yields + // *is* the table address and no low-12-bit add is needed. + let jump_table_label = self.jump_table_label; + self.push(a64::adrp(SCRATCH1, jump_table_label)); + self.push(a64::add_reg(X, SCRATCH1, SCRATCH1, SCRATCH2)); + self.push(a64::ldr_imm(a64::Size::U64, SCRATCH1, SCRATCH1, 0)); + if let Some((ra, value)) = load_imm { + self.load_imm(ra, value as i32); + } + self.push(a64::br(SCRATCH1)); + } + + /// Branches to a (possibly not-yet-defined) label, choosing the widest-range encoding needed. + fn jump_to_label(&mut self, label: Label) { + self.push(a64::b(label)); + } + + /// Conditional branch to `label`: inverted `B.cond` over an unconditional `B` (±128MB range). + fn branch_to_label(&mut self, cond: a64::Cond, label: Label) { + let skip = self.0.asm.forward_declare_label(); + self.push(a64::b_cond(cond.invert(), skip)); + self.push(a64::b(label)); + self.0.asm.define_label(skip); + } + + /// Resolve a jump target to its label. An invalid target (e.g. mid-block) has no defined label and + /// would patch to a branch-to-self, so route it to the trap trampoline (traps when reached). + fn jump_target_label(&mut self, target: u32) -> Label { + if self.is_jump_target_valid(target) { + self.get_or_forward_declare_label(target).unwrap_or(self.trap_label) + } else { + self.trap_label + } + } + + /// `cmp s1, s2; b. target` (register/register form). `cond` is in terms of `s1 s2`. + fn branch_rr(&mut self, s1: RawReg, s2: RawReg, target: u32, cond: a64::Cond) { + if !self.is_jump_target_valid(target) { + // Invalid target: the instruction traps unconditionally (matches the interpreter), whether + // or not the branch would be taken. + let trap = self.trap_label; + self.jump_to_label(trap); + return; + } + let sz = self.reg_size(); + self.push(a64::cmp_reg(sz, conv_reg(s1), conv_reg(s2))); + let label = self.get_or_forward_declare_label(target).unwrap_or(self.trap_label); + self.branch_to_label(cond, label); + } + + /// `cmp s1, #imm; b. target` (register/immediate form). + fn branch_ri(&mut self, s1: RawReg, imm: u32, target: u32, cond: a64::Cond) { + if !self.is_jump_target_valid(target) { + let trap = self.trap_label; + self.jump_to_label(trap); + return; + } + let sz = self.reg_size(); + self.cmp_const(sz, conv_reg(s1), self.imm_operand(imm)); + let label = self.get_or_forward_declare_label(target).unwrap_or(self.trap_label); + self.branch_to_label(cond, label); + } + + pub fn add_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::add_reg(a64::RegSize::W, conv_reg(d), conv_reg(s1), conv_reg(s2))); + self.finish32(d); + } + pub fn add_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::add_reg(a64::RegSize::X, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn add_imm_32(&mut self, d: RawReg, s1: RawReg, s2: i32) { + self.add_const_sized(a64::RegSize::W, conv_reg(d), conv_reg(s1), i64::from(s2)); + self.finish32(d); + } + pub fn add_imm_64(&mut self, d: RawReg, s1: RawReg, s2: i32) { + // The immediate is sign-extended to 64 bits (as in the x86 backend's `imm64(sign_extend)`). + self.add_const_sized(a64::RegSize::X, conv_reg(d), conv_reg(s1), i64::from(s2)); + } + pub fn and(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + let sz = self.reg_size(); + self.push(a64::and_reg(sz, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn and_imm(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let s2 = s2 as u32; + let sz = self.reg_size(); + if let Some(imm) = self.logical_imm(s2) { + self.push(a64::and_imm(sz, conv_reg(d), conv_reg(s1), imm)); + } else { + let r = self.imm_reg(s2); + self.push(a64::and_reg(sz, conv_reg(d), conv_reg(s1), r)); + } + } + pub fn and_inverted(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + // d = s1 & ~s2 + let sz = self.reg_size(); + self.push(a64::bic_reg(sz, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn branch_eq(&mut self, s1: RawReg, s2: RawReg, target: u32) { + self.branch_rr(s1, s2, target, a64::Cond::Eq); + } + pub fn branch_eq_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Eq); + } + pub fn branch_not_eq(&mut self, s1: RawReg, s2: RawReg, target: u32) { + self.branch_rr(s1, s2, target, a64::Cond::Ne); + } + pub fn branch_not_eq_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Ne); + } + pub fn branch_less_unsigned(&mut self, s1: RawReg, s2: RawReg, target: u32) { + self.branch_rr(s1, s2, target, a64::Cond::Cc); + } + pub fn branch_less_unsigned_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Cc); + } + pub fn branch_less_signed(&mut self, s1: RawReg, s2: RawReg, target: u32) { + self.branch_rr(s1, s2, target, a64::Cond::Lt); + } + pub fn branch_less_signed_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Lt); + } + pub fn branch_greater_or_equal_unsigned(&mut self, s1: RawReg, s2: RawReg, target: u32) { + self.branch_rr(s1, s2, target, a64::Cond::Cs); + } + pub fn branch_greater_or_equal_unsigned_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Cs); + } + pub fn branch_greater_or_equal_signed(&mut self, s1: RawReg, s2: RawReg, target: u32) { + self.branch_rr(s1, s2, target, a64::Cond::Ge); + } + pub fn branch_greater_or_equal_signed_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Ge); + } + pub fn branch_greater_unsigned_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Hi); + } + pub fn branch_greater_signed_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Gt); + } + pub fn branch_less_or_equal_unsigned_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Ls); + } + pub fn branch_less_or_equal_signed_imm(&mut self, s1: RawReg, s2: i32, target: u32) { + let s2 = s2 as u32; + self.branch_ri(s1, s2, target, a64::Cond::Le); + } + pub fn cmov_if_not_zero(&mut self, d: RawReg, s: RawReg, c: RawReg) { + let sz = self.reg_size(); + self.push(a64::cmp_imm(sz, conv_reg(c), 0, false)); + self.push(a64::csel(sz, conv_reg(d), conv_reg(s), conv_reg(d), a64::Cond::Ne)); + } + pub fn cmov_if_not_zero_imm(&mut self, d: RawReg, c: RawReg, s: i32) { + let s = s as u32; + let sz = self.reg_size(); + let r = self.imm_reg(s); + self.push(a64::cmp_imm(sz, conv_reg(c), 0, false)); + self.push(a64::csel(sz, conv_reg(d), r, conv_reg(d), a64::Cond::Ne)); + } + pub fn cmov_if_zero(&mut self, d: RawReg, s: RawReg, c: RawReg) { + let sz = self.reg_size(); + self.push(a64::cmp_imm(sz, conv_reg(c), 0, false)); + self.push(a64::csel(sz, conv_reg(d), conv_reg(s), conv_reg(d), a64::Cond::Eq)); + } + pub fn cmov_if_zero_imm(&mut self, d: RawReg, c: RawReg, s: i32) { + let s = s as u32; + let sz = self.reg_size(); + let r = self.imm_reg(s); + self.push(a64::cmp_imm(sz, conv_reg(c), 0, false)); + self.push(a64::csel(sz, conv_reg(d), r, conv_reg(d), a64::Cond::Eq)); + } + pub fn count_leading_zero_bits_32(&mut self, d: RawReg, s: RawReg) { + self.push(a64::clz(a64::RegSize::W, conv_reg(d), conv_reg(s))); + self.finish32(d); + } + pub fn count_leading_zero_bits_64(&mut self, d: RawReg, s: RawReg) { + self.push(a64::clz(a64::RegSize::X, conv_reg(d), conv_reg(s))); + } + pub fn count_trailing_zero_bits_32(&mut self, d: RawReg, s: RawReg) { + // ctz(x) = clz(rbit(x)) + self.push(a64::rbit(a64::RegSize::W, conv_reg(d), conv_reg(s))); + self.push(a64::clz(a64::RegSize::W, conv_reg(d), conv_reg(d))); + self.finish32(d); + } + pub fn count_trailing_zero_bits_64(&mut self, d: RawReg, s: RawReg) { + self.push(a64::rbit(a64::RegSize::X, conv_reg(d), conv_reg(s))); + self.push(a64::clz(a64::RegSize::X, conv_reg(d), conv_reg(d))); + } + // popcount via NEON (no scalar instruction): move to v0, count per byte, sum. Result is small and + // positive, so no 32-bit sign-extension is needed. + fn emit_popcount(&mut self, d: RawReg, s: RawReg, is_64: bool) { + let d = conv_reg(d); + let s = conv_reg(s); + if is_64 { + self.push(a64::fmov_gpr_to_d(0, s)); + } else { + self.push(a64::fmov_gpr_to_s(0, s)); + } + self.push(a64::cnt_8b(0, 0)); + self.push(a64::addv_8b(0, 0)); + self.push(a64::fmov_s_to_gpr(d, 0)); + } + pub fn count_set_bits_32(&mut self, d: RawReg, s: RawReg) { + self.emit_popcount(d, s, false); + } + pub fn count_set_bits_64(&mut self, d: RawReg, s: RawReg) { + self.emit_popcount(d, s, true); + } + // RISC-V vs AArch64 differ only on divide-by-zero (RISC-V: quotient -1, remainder = dividend). + // `msub` already yields the dividend when the quotient is 0; only the quotient needs the -1 fix-up. + fn emit_div(&mut self, d: RawReg, s1: RawReg, s2: RawReg, size: a64::RegSize, signed: bool) { + let d = conv_reg(d); + let s1 = conv_reg(s1); + let s2 = conv_reg(s2); + if signed { + self.push(a64::sdiv(size, SCRATCH0, s1, s2)); + } else { + self.push(a64::udiv(size, SCRATCH0, s1, s2)); + } + self.push(a64::cmp_imm(size, s2, 0, false)); + // s2 != 0 ? quotient : ~xzr (= -1) + self.push(a64::csinv(size, d, SCRATCH0, NativeReg::XZR, a64::Cond::Ne)); + } + + fn emit_rem(&mut self, d: RawReg, s1: RawReg, s2: RawReg, size: a64::RegSize, signed: bool) { + let d = conv_reg(d); + let s1 = conv_reg(s1); + let s2 = conv_reg(s2); + if signed { + self.push(a64::sdiv(size, SCRATCH0, s1, s2)); + } else { + self.push(a64::udiv(size, SCRATCH0, s1, s2)); + } + // remainder = s1 - quotient * s2; on a zero divisor the quotient is 0, so this yields s1. + self.push(a64::msub(size, d, SCRATCH0, s2, s1)); + } + + pub fn div_signed_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.emit_div(d, s1, s2, a64::RegSize::W, true); + self.finish32(d); + } + pub fn div_signed_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.emit_div(d, s1, s2, a64::RegSize::X, true); + } + pub fn div_unsigned_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.emit_div(d, s1, s2, a64::RegSize::W, false); + self.finish32(d); + } + pub fn div_unsigned_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.emit_div(d, s1, s2, a64::RegSize::X, false); + } + pub fn ecalli(&mut self, code_offset: u32, args_length: u32, imm: i32) { + let imm = imm as u32; + if let Some(ref custom_codegen) = self.0.custom_codegen { + if !custom_codegen.should_emit_ecalli(imm, &mut self.0.asm) { + return; + } + } + self.st_vmctx_imm32(imm, S::offset_table().arg); + self.st_vmctx_imm32(code_offset, S::offset_table().program_counter); + self.st_vmctx_imm32(code_offset + args_length + 1, S::offset_table().next_program_counter); + let ecall_label = self.ecall_label; + self.push(a64::bl(ecall_label)); + } + pub(crate) fn emit_divrem_trampoline(&mut self) { + // div/rem are handled inline; these labels are unreachable but defined so they aren't dangling. + let labels = [ + self.div32u_label, + self.div32s_label, + self.div64u_label, + self.div64s_label, + self.rem32u_label, + self.rem32s_label, + self.rem64u_label, + self.rem64s_label, + ]; + for label in labels { + self.0.asm.define_label(label); + self.push(a64::udf(0)); + } + } + pub(crate) fn emit_ecall_trampoline(&mut self) { + let label = self.ecall_label; + self.0.asm.define_label(label); + self.save_return_address_to_vmctx(); + self.save_registers_to_vmctx(); + self.mov_imm(TMP_REG, S::address_table().syscall_hostcall); + self.push(a64::br(TMP_REG)); + } + pub(crate) fn emit_gas_metering_stub(&mut self, kind: GasMeteringKind) { + let origin = self.0.asm.len(); + assert_eq!(S::offset_table().gas, 0x60); + + // Gas counter sits a fixed displacement below the memory base: one `sub` immediate finds it. + let disp = -(crate::sandbox::GUEST_MEMORY_TO_VMCTX_OFFSET as i64 + S::offset_table().gas as i64); + debug_assert!(disp > 0 && disp < 0x1000, "gas displacement does not fit a sub-immediate"); + + // Fixed layout (offsets fixed by GAS_COST_OFFSET / GAS_METERING_TRAP_OFFSET). + self.0.asm.push_raw(&0x1800_0050u32.to_le_bytes()); // +0 ldr w16, #8 (load cost) + self.0.asm.push_raw(&0x1400_0002u32.to_le_bytes()); // +4 b #8 (skip literal) + self.0.asm.push_raw(&i32::MAX.to_le_bytes()); // +8 cost literal (patched by emit_weight) + self.push(a64::sub_imm(a64::RegSize::X, SCRATCH1, MEMORY_BASE_REG, disp as u32, false)); // +12 &gas + self.push(a64::ldr_imm(a64::Size::U64, SCRATCH2, SCRATCH1, 0)); // +16 gas + self.push(a64::sub_reg(a64::RegSize::X, SCRATCH2, SCRATCH2, SCRATCH0)); // +20 gas -= cost + self.push(a64::str_imm(a64::Size::U64, SCRATCH2, SCRATCH1, 0)); // +24 + + if matches!(kind, GasMeteringKind::Sync) { + self.push(a64::cmp_imm(a64::RegSize::X, SCRATCH2, 0, false)); // +28 + self.0.asm.push_raw(&0x5400_004Au32.to_le_bytes()); // +32 b.ge #8 (skip trap if gas >= 0) + self.push(a64::udf(0)); // +36 underflow -> SIGILL + debug_assert_eq!(GAS_METERING_TRAP_OFFSET, (self.0.asm.len() - origin - 4) as u64); + } + debug_assert_eq!(GAS_COST_OFFSET, 8); + } + pub(crate) fn emit_memset_trampoline(&mut self) { + // Out-of-gas path for `memset`: A0/A2 already reflect progress, so just report NotEnoughGas. + let label = self.memset_label; + self.0.asm.define_label(label); + self.save_registers_to_vmctx(); + self.mov_imm(TMP_REG, S::address_table().syscall_not_enough_gas); + self.push(a64::br(TMP_REG)); + } + pub(crate) fn emit_sbrk_trampoline(&mut self) { + let label = self.sbrk_label; + self.0.asm.define_label(label); + + // Arg in TMP_REG, return addr in x30. Save x30 across the C call to syscall_sbrk; result in x0. + self.push(a64::sub_imm(a64::RegSize::X, SP, SP, 16, false)); + self.push(a64::str_imm(a64::Size::U64, NativeReg::X30, SP, 0)); + self.push(a64::mov_reg(a64::RegSize::X, SCRATCH2, TMP_REG)); // preserve arg across save_registers + self.save_registers_to_vmctx(); + self.push(a64::mov_reg(a64::RegSize::X, NativeReg::X0, SCRATCH2)); // arg -> x0 + self.mov_imm(TMP_REG, S::address_table().syscall_sbrk); + self.push(a64::blr(TMP_REG)); + self.push(a64::mov_reg(a64::RegSize::X, SCRATCH2, NativeReg::X0)); // stash result + self.restore_registers_from_vmctx(); + self.push(a64::mov_reg(a64::RegSize::X, TMP_REG, SCRATCH2)); // result -> TMP_REG for the handler + self.push(a64::ldr_imm(a64::Size::U64, NativeReg::X30, SP, 0)); + self.push(a64::add_imm(a64::RegSize::X, SP, SP, 16, false)); + self.push(a64::ret(NativeReg::X30)); + } + pub(crate) fn emit_step_trampoline(&mut self) { + let label = self.step_label; + self.0.asm.define_label(label); + self.save_return_address_to_vmctx(); + self.save_registers_to_vmctx(); + self.mov_imm(TMP_REG, S::address_table().syscall_step); + self.push(a64::br(TMP_REG)); + } + pub(crate) fn emit_sysenter(&mut self) -> Label { + let label = self.0.asm.create_label(); + self.restore_registers_from_vmctx(); + self.ld_vmctx(TMP_REG, S::offset_table().next_native_program_counter); + self.push(a64::br(TMP_REG)); + label + } + + pub(crate) fn emit_sysreturn(&mut self) -> Label { + let label = self.0.asm.create_label(); + self.st_vmctx_imm(0, S::offset_table().next_native_program_counter); + self.save_registers_to_vmctx(); + self.mov_imm(TMP_REG, S::address_table().syscall_return); + self.push(a64::br(TMP_REG)); + label + } + pub(crate) fn emit_trap_trampoline(&mut self) { + let label = self.trap_label; + self.0.asm.define_label(label); + self.save_registers_to_vmctx(); + self.st_vmctx_imm(0, S::offset_table().next_native_program_counter); + self.mov_imm(TMP_REG, S::address_table().syscall_trap); + self.push(a64::br(TMP_REG)); + } + pub(crate) fn emit_weight(&mut self, offset: usize, cost: u32) { + // Patch the gas-cost literal embedded in the metering stub at `offset`. + let p = offset + GAS_COST_OFFSET; + self.0.asm.code_mut()[p..p + 4].copy_from_slice(&cost.to_le_bytes()); + } + #[allow(clippy::unused_self)] + #[allow(clippy::needless_pass_by_ref_mut)] + pub fn fallthrough(&mut self) {} + pub fn invalid(&mut self, code_offset: u32) { + log::debug!("Encountered invalid instruction"); + self.trap(code_offset); + } + pub fn jump(&mut self, target: u32) { + let label = self.jump_target_label(target); + self.jump_to_label(label); + } + pub fn jump_indirect(&mut self, base: RawReg, offset: i32) { + let offset = offset as u32; + self.jump_indirect_impl(None, base, offset); + } + pub fn load_u8(&mut self, dst: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, None, offset, a64::Size::U8, false); + } + pub fn load_i8(&mut self, dst: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, None, offset, a64::Size::U8, true); + } + pub fn load_u16(&mut self, dst: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, None, offset, a64::Size::U16, false); + } + pub fn load_i16(&mut self, dst: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, None, offset, a64::Size::U16, true); + } + pub fn load_u32(&mut self, dst: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, None, offset, a64::Size::U32, false); + } + pub fn load_i32(&mut self, dst: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, None, offset, a64::Size::U32, true); + } + pub fn load_u64(&mut self, dst: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, None, offset, a64::Size::U64, false); + } + pub fn load_indirect_u8(&mut self, dst: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, Some(base), offset, a64::Size::U8, false); + } + pub fn load_indirect_i8(&mut self, dst: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, Some(base), offset, a64::Size::U8, true); + } + pub fn load_indirect_u16(&mut self, dst: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, Some(base), offset, a64::Size::U16, false); + } + pub fn load_indirect_i16(&mut self, dst: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, Some(base), offset, a64::Size::U16, true); + } + pub fn load_indirect_u32(&mut self, dst: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, Some(base), offset, a64::Size::U32, false); + } + pub fn load_indirect_i32(&mut self, dst: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, Some(base), offset, a64::Size::U32, true); + } + pub fn load_indirect_u64(&mut self, dst: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_load(dst, Some(base), offset, a64::Size::U64, false); + } + pub fn load_imm(&mut self, dst: RawReg, s2: i32) { + let s2 = s2 as u32; + let value = match B::BITNESS { + Bitness::B32 => u64::from(s2), + Bitness::B64 => i64::from(s2 as i32) as u64, + }; + self.mov_imm(conv_reg(dst), value); + } + pub fn load_imm_and_jump(&mut self, ra: RawReg, value: i32, target: u32) { + self.load_imm(ra, value); + let label = self.jump_target_label(target); + self.jump_to_label(label); + } + pub fn load_imm_and_jump_indirect(&mut self, ra: RawReg, base: RawReg, value: i32, offset: i32) { + let value = value as u32; + let offset = offset as u32; + self.jump_indirect_impl(Some((ra, value)), base, offset); + } + pub fn load_imm64(&mut self, dst: RawReg, s2: u64) { + self.mov_imm(conv_reg(dst), s2); + } + pub fn maximum(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + let sz = self.reg_size(); + self.push(a64::cmp_reg(sz, conv_reg(s1), conv_reg(s2))); + self.push(a64::csel(sz, conv_reg(d), conv_reg(s1), conv_reg(s2), a64::Cond::Gt)); + } + pub fn maximum_unsigned(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + let sz = self.reg_size(); + self.push(a64::cmp_reg(sz, conv_reg(s1), conv_reg(s2))); + self.push(a64::csel(sz, conv_reg(d), conv_reg(s1), conv_reg(s2), a64::Cond::Hi)); + } + pub fn minimum(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + let sz = self.reg_size(); + self.push(a64::cmp_reg(sz, conv_reg(s1), conv_reg(s2))); + self.push(a64::csel(sz, conv_reg(d), conv_reg(s1), conv_reg(s2), a64::Cond::Lt)); + } + pub fn minimum_unsigned(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + let sz = self.reg_size(); + self.push(a64::cmp_reg(sz, conv_reg(s1), conv_reg(s2))); + self.push(a64::csel(sz, conv_reg(d), conv_reg(s1), conv_reg(s2), a64::Cond::Cc)); + } + pub fn memset(&mut self) { + // Fill A2 bytes at A0 with A1's low byte, 1 gas/byte; A0/A2 update in place to track progress on fault/exhaustion. + use a64::RegSize::{W, X}; + let dst = conv_reg(Reg::A0.into()); + let val = conv_reg(Reg::A1.into()); + let cnt = conv_reg(Reg::A2.into()); + let metering = self.gas_metering.is_some(); + + let memset_label = self.memset_label; + let sz = self.reg_size(); + // The count is always 32-bit (the interpreter reads A2 as u32); clear its upper bits so a + // dirty A2 doesn't run an over-long fill and so A2 is written back zero-extended on exit. + self.push(a64::mov_reg(W, cnt, cnt)); + // 32-bit mode: clear A0's upper bits for the address calc. 64-bit mode: A0 is a full pointer, leave it. + if matches!(B::BITNESS, Bitness::B32) { + self.push(a64::mov_reg(W, dst, dst)); + } + + let done = self.0.asm.forward_declare_label(); + let loop_top = self.0.asm.create_label(); + self.push(a64::cbz(X, cnt, done)); + if metering { + // On gas exhaustion branch to the (shared) trampoline, which reports NotEnoughGas. + self.ld_vmctx(SCRATCH2, S::offset_table().gas); + let has_gas = self.0.asm.forward_declare_label(); + self.push(a64::cbnz(X, SCRATCH2, has_gas)); + self.push(a64::b(memset_label)); + self.0.asm.define_label(has_gas); + } + // Guest addresses are 32-bit; use A0's low 32 bits (matches the interpreter's u32 destination). + self.push(a64::mov_reg(W, SCRATCH0, dst)); + self.push(a64::add_reg(X, SCRATCH0, MEMORY_BASE_REG, SCRATCH0)); + self.push(a64::str_imm(a64::Size::U8, val, SCRATCH0, 0)); + if metering { + self.push(a64::sub_imm(X, SCRATCH2, SCRATCH2, 1, false)); + self.st_vmctx(SCRATCH2, S::offset_table().gas); + } + self.push(a64::add_imm(sz, dst, dst, 1, false)); + self.push(a64::sub_imm(X, cnt, cnt, 1, false)); + self.jump_to_label(loop_top); + self.0.asm.define_label(done); + } + pub fn move_reg(&mut self, d: RawReg, s: RawReg) { + let sz = self.reg_size(); + self.push(a64::mov_reg(sz, conv_reg(d), conv_reg(s))); + } + pub fn mul_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::madd(a64::RegSize::W, conv_reg(d), conv_reg(s1), conv_reg(s2), NativeReg::XZR)); + self.finish32(d); + } + pub fn mul_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::madd(a64::RegSize::X, conv_reg(d), conv_reg(s1), conv_reg(s2), NativeReg::XZR)); + } + pub fn mul_imm_32(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let s2 = s2 as u32; + let r = self.imm_reg(s2); + self.push(a64::madd(a64::RegSize::W, conv_reg(d), conv_reg(s1), r, NativeReg::XZR)); + self.finish32(d); + } + pub fn mul_imm_64(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let s2 = s2 as u32; + self.mov_imm(SCRATCH0, i64::from(s2 as i32) as u64); + self.push(a64::madd(a64::RegSize::X, conv_reg(d), conv_reg(s1), SCRATCH0, NativeReg::XZR)); + } + pub fn mul_upper_signed_signed(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + use a64::RegSize::X; + let d = conv_reg(d); + let s1 = conv_reg(s1); + let s2 = conv_reg(s2); + if matches!(B::BITNESS, Bitness::B32) { + // High 32 of a signed 32x32 product: widen (sxtw), 64-bit multiply, take bits [63:32]. + self.push(a64::sbfm(X, SCRATCH0, s1, 0, 31)); + self.push(a64::sbfm(X, SCRATCH1, s2, 0, 31)); + self.push(a64::madd(X, d, SCRATCH0, SCRATCH1, NativeReg::XZR)); + self.push(a64::ubfm(X, d, d, 32, 63)); // lsr #32 + } else { + self.push(a64::smulh(d, s1, s2)); + } + } + pub fn mul_upper_unsigned_unsigned(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + use a64::RegSize::X; + let d = conv_reg(d); + let s1 = conv_reg(s1); + let s2 = conv_reg(s2); + if matches!(B::BITNESS, Bitness::B32) { + // High 32 of an unsigned 32x32 product: widen (uxtw), 64-bit multiply, take bits [63:32]. + self.push(a64::ubfm(X, SCRATCH0, s1, 0, 31)); + self.push(a64::ubfm(X, SCRATCH1, s2, 0, 31)); + self.push(a64::madd(X, d, SCRATCH0, SCRATCH1, NativeReg::XZR)); + self.push(a64::ubfm(X, d, d, 32, 63)); // lsr #32 + } else { + self.push(a64::umulh(d, s1, s2)); + } + } + pub fn mul_upper_signed_unsigned(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + use a64::RegSize::X; + let d = conv_reg(d); + let s1 = conv_reg(s1); + let s2 = conv_reg(s2); + if matches!(B::BITNESS, Bitness::B32) { + // High 32 of signed(s1) * unsigned(s2): sxtw s1, uxtw s2, 64-bit multiply, bits [63:32]. + self.push(a64::sbfm(X, SCRATCH0, s1, 0, 31)); + self.push(a64::ubfm(X, SCRATCH1, s2, 0, 31)); + self.push(a64::madd(X, d, SCRATCH0, SCRATCH1, NativeReg::XZR)); + self.push(a64::ubfm(X, d, d, 32, 63)); // lsr #32 + } else { + // High64 of signed(s1) * unsigned(s2) = umulh(s1,s2) - (s1<0 ? s2 : 0). + self.push(a64::umulh(SCRATCH0, s1, s2)); // SCRATCH0 = unsigned high + self.push(a64::sbfm(X, SCRATCH1, s1, 63, 63)); // asr #63: sign mask + self.push(a64::and_reg(X, SCRATCH1, SCRATCH1, s2)); + self.push(a64::sub_reg(X, d, SCRATCH0, SCRATCH1)); + } + } + pub fn negate_and_add_imm_32(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let s2 = s2 as u32; + // d = s2 - s1 + let r = self.imm_reg(s2); + self.push(a64::sub_reg(a64::RegSize::W, conv_reg(d), r, conv_reg(s1))); + self.finish32(d); + } + pub fn negate_and_add_imm_64(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let s2 = s2 as u32; + self.mov_imm(SCRATCH0, i64::from(s2 as i32) as u64); + self.push(a64::sub_reg(a64::RegSize::X, conv_reg(d), SCRATCH0, conv_reg(s1))); + } + pub fn or(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + let sz = self.reg_size(); + self.push(a64::orr_reg(sz, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn or_imm(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let s2 = s2 as u32; + let sz = self.reg_size(); + if let Some(imm) = self.logical_imm(s2) { + self.push(a64::orr_imm(sz, conv_reg(d), conv_reg(s1), imm)); + } else { + let r = self.imm_reg(s2); + self.push(a64::orr_reg(sz, conv_reg(d), conv_reg(s1), r)); + } + } + pub fn or_inverted(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + // d = s1 | ~s2 + let sz = self.reg_size(); + self.push(a64::orn_reg(sz, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn rem_signed_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.emit_rem(d, s1, s2, a64::RegSize::W, true); + self.finish32(d); + } + pub fn rem_signed_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.emit_rem(d, s1, s2, a64::RegSize::X, true); + } + pub fn rem_unsigned_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.emit_rem(d, s1, s2, a64::RegSize::W, false); + self.finish32(d); + } + pub fn rem_unsigned_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.emit_rem(d, s1, s2, a64::RegSize::X, false); + } + pub fn reverse_byte(&mut self, d: RawReg, s: RawReg) { + match B::BITNESS { + Bitness::B32 => self.push(a64::rev32_word(conv_reg(d), conv_reg(s))), + Bitness::B64 => self.push(a64::rev64(conv_reg(d), conv_reg(s))), + } + } + pub fn rotate_left_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + // rol(x, n) = ror(x, -n); AArch64 only has ror. + self.push(a64::sub_reg(a64::RegSize::W, SCRATCH0, NativeReg::XZR, conv_reg(s2))); + self.push(a64::rorv(a64::RegSize::W, conv_reg(d), conv_reg(s1), SCRATCH0)); + self.finish32(d); + } + pub fn rotate_left_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::sub_reg(a64::RegSize::X, SCRATCH0, NativeReg::XZR, conv_reg(s2))); + self.push(a64::rorv(a64::RegSize::X, conv_reg(d), conv_reg(s1), SCRATCH0)); + } + pub fn rotate_right_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::rorv(a64::RegSize::W, conv_reg(d), conv_reg(s1), conv_reg(s2))); + self.finish32(d); + } + pub fn rotate_right_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::rorv(a64::RegSize::X, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn rotate_right_imm_32(&mut self, d: RawReg, s: RawReg, c: i32) { + // `ror rd, rn, #n` is `extr rd, rn, rn, #n`. + let n = c as u32 & 31; + let s = conv_reg(s); + self.push(a64::extr(a64::RegSize::W, conv_reg(d), s, s, n)); + self.finish32(d); + } + pub fn rotate_right_imm_64(&mut self, d: RawReg, s: RawReg, c: i32) { + let n = c as u32 & 63; + let s = conv_reg(s); + self.push(a64::extr(a64::RegSize::X, conv_reg(d), s, s, n)); + } + pub fn rotate_right_imm_alt_32(&mut self, d: RawReg, s: RawReg, c: i32) { + let c = c as u32; + // d = ror(c, s): the immediate is the value, the register is the amount. + let r = self.imm_reg(c); + self.push(a64::rorv(a64::RegSize::W, conv_reg(d), r, conv_reg(s))); + self.finish32(d); + } + pub fn rotate_right_imm_alt_64(&mut self, d: RawReg, s: RawReg, c: i32) { + let c = c as u32; + self.mov_imm(SCRATCH0, i64::from(c as i32) as u64); + self.push(a64::rorv(a64::RegSize::X, conv_reg(d), SCRATCH0, conv_reg(s))); + } + pub fn sbrk(&mut self, dst: RawReg, size: RawReg) { + use a64::RegSize::{W, X}; + // pending heap top = heap_top + size; if it stays within the threshold just bump the pointer, + // otherwise call the trampoline (which maps memory or returns 0). Mirrors the x86 backend. + let dst = conv_reg(dst); + let size = conv_reg(size); + let heap = S::offset_table().heap_info; + + self.push(a64::mov_reg(W, SCRATCH1, size)); // zero-extend the 32-bit size + self.ld_vmctx(SCRATCH2, heap); // heap_top + self.push(a64::add_reg(X, SCRATCH2, SCRATCH2, SCRATCH1)); // pending = heap_top + size + self.ld_vmctx(SCRATCH1, heap + 8); // heap_threshold + self.push(a64::cmp_reg(X, SCRATCH2, SCRATCH1)); + + let bump_only = self.0.asm.forward_declare_label(); + let cont = self.0.asm.forward_declare_label(); + self.branch_to_label(a64::Cond::Ls, bump_only); // pending <= threshold + + // Crossed the threshold: the trampoline maps memory and returns the new top (or 0) in TMP_REG. + self.push(a64::mov_reg(X, TMP_REG, SCRATCH2)); + let sbrk_label = self.sbrk_label; + self.push(a64::bl(sbrk_label)); + self.push(a64::mov_reg(X, dst, TMP_REG)); + self.jump_to_label(cont); + + self.0.asm.define_label(bump_only); + self.st_vmctx(SCRATCH2, heap); // heap_top = pending + self.push(a64::mov_reg(X, dst, SCRATCH2)); + self.0.asm.define_label(cont); + } + // `cset d, cond` is `csinc d, xzr, xzr, invert(cond)`; we pass the already-inverted condition. + fn set_cond(&mut self, d: RawReg, cond_when_true: a64::Cond) { + let sz = self.reg_size(); + self.push(a64::csinc(sz, conv_reg(d), NativeReg::XZR, NativeReg::XZR, cond_when_true.invert())); + } + + pub fn set_less_than_unsigned(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + let sz = self.reg_size(); + self.push(a64::cmp_reg(sz, conv_reg(s1), conv_reg(s2))); + self.set_cond(d, a64::Cond::Cc); + } + pub fn set_less_than_signed(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + let sz = self.reg_size(); + self.push(a64::cmp_reg(sz, conv_reg(s1), conv_reg(s2))); + self.set_cond(d, a64::Cond::Lt); + } + pub fn set_less_than_unsigned_imm(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let sz = self.reg_size(); + self.cmp_const(sz, conv_reg(s1), self.imm_operand(s2 as u32)); + self.set_cond(d, a64::Cond::Cc); + } + pub fn set_less_than_signed_imm(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let sz = self.reg_size(); + self.cmp_const(sz, conv_reg(s1), self.imm_operand(s2 as u32)); + self.set_cond(d, a64::Cond::Lt); + } + pub fn set_greater_than_unsigned_imm(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let sz = self.reg_size(); + self.cmp_const(sz, conv_reg(s1), self.imm_operand(s2 as u32)); + self.set_cond(d, a64::Cond::Hi); + } + pub fn set_greater_than_signed_imm(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let sz = self.reg_size(); + self.cmp_const(sz, conv_reg(s1), self.imm_operand(s2 as u32)); + self.set_cond(d, a64::Cond::Gt); + } + pub fn shift_arithmetic_right_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::asrv(a64::RegSize::W, conv_reg(d), conv_reg(s1), conv_reg(s2))); + self.finish32(d); + } + pub fn shift_arithmetic_right_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::asrv(a64::RegSize::X, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn shift_arithmetic_right_imm_32(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let n = s2 as u32 & 31; + self.push(a64::sbfm(a64::RegSize::W, conv_reg(d), conv_reg(s1), n, 31)); + self.finish32(d); + } + pub fn shift_arithmetic_right_imm_64(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let n = s2 as u32 & 63; + self.push(a64::sbfm(a64::RegSize::X, conv_reg(d), conv_reg(s1), n, 63)); + } + pub fn shift_arithmetic_right_imm_alt_32(&mut self, d: RawReg, s2: RawReg, s1: i32) { + let s1 = s1 as u32; + let r = self.imm_reg(s1); + self.push(a64::asrv(a64::RegSize::W, conv_reg(d), r, conv_reg(s2))); + self.finish32(d); + } + pub fn shift_arithmetic_right_imm_alt_64(&mut self, d: RawReg, s2: RawReg, s1: i32) { + let s1 = s1 as u32; + self.mov_imm(SCRATCH0, i64::from(s1 as i32) as u64); + self.push(a64::asrv(a64::RegSize::X, conv_reg(d), SCRATCH0, conv_reg(s2))); + } + pub fn shift_logical_left_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::lslv(a64::RegSize::W, conv_reg(d), conv_reg(s1), conv_reg(s2))); + self.finish32(d); + } + pub fn shift_logical_left_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::lslv(a64::RegSize::X, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn shift_logical_left_imm_32(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let n = s2 as u32 & 31; + self.shl_imm(a64::RegSize::W, conv_reg(d), conv_reg(s1), n); + self.finish32(d); + } + pub fn shift_logical_left_imm_64(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let n = s2 as u32 & 63; + self.shl_imm(a64::RegSize::X, conv_reg(d), conv_reg(s1), n); + } + pub fn shift_logical_left_imm_alt_32(&mut self, d: RawReg, s2: RawReg, s1: i32) { + let s1 = s1 as u32; + let r = self.imm_reg(s1); + self.push(a64::lslv(a64::RegSize::W, conv_reg(d), r, conv_reg(s2))); + self.finish32(d); + } + pub fn shift_logical_left_imm_alt_64(&mut self, d: RawReg, s2: RawReg, s1: i32) { + let s1 = s1 as u32; + self.mov_imm(SCRATCH0, i64::from(s1 as i32) as u64); + self.push(a64::lslv(a64::RegSize::X, conv_reg(d), SCRATCH0, conv_reg(s2))); + } + pub fn shift_logical_right_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::lsrv(a64::RegSize::W, conv_reg(d), conv_reg(s1), conv_reg(s2))); + self.finish32(d); + } + pub fn shift_logical_right_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::lsrv(a64::RegSize::X, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn shift_logical_right_imm_32(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let n = s2 as u32 & 31; + self.push(a64::ubfm(a64::RegSize::W, conv_reg(d), conv_reg(s1), n, 31)); + self.finish32(d); + } + pub fn shift_logical_right_imm_64(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let n = s2 as u32 & 63; + self.push(a64::ubfm(a64::RegSize::X, conv_reg(d), conv_reg(s1), n, 63)); + } + pub fn shift_logical_right_imm_alt_32(&mut self, d: RawReg, s2: RawReg, s1: i32) { + let s1 = s1 as u32; + let r = self.imm_reg(s1); + self.push(a64::lsrv(a64::RegSize::W, conv_reg(d), r, conv_reg(s2))); + self.finish32(d); + } + pub fn shift_logical_right_imm_alt_64(&mut self, d: RawReg, s2: RawReg, s1: i32) { + let s1 = s1 as u32; + self.mov_imm(SCRATCH0, i64::from(s1 as i32) as u64); + self.push(a64::lsrv(a64::RegSize::X, conv_reg(d), SCRATCH0, conv_reg(s2))); + } + pub fn sign_extend_8(&mut self, d: RawReg, s: RawReg) { + let sz = self.reg_size(); + self.push(a64::sbfm(sz, conv_reg(d), conv_reg(s), 0, 7)); + } + pub fn sign_extend_16(&mut self, d: RawReg, s: RawReg) { + let sz = self.reg_size(); + self.push(a64::sbfm(sz, conv_reg(d), conv_reg(s), 0, 15)); + } + pub fn store_u8(&mut self, src: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_store_reg(src, None, offset, a64::Size::U8); + } + pub fn store_u16(&mut self, src: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_store_reg(src, None, offset, a64::Size::U16); + } + pub fn store_u32(&mut self, src: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_store_reg(src, None, offset, a64::Size::U32); + } + pub fn store_u64(&mut self, src: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_store_reg(src, None, offset, a64::Size::U64); + } + pub fn store_indirect_u8(&mut self, src: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_store_reg(src, Some(base), offset, a64::Size::U8); + } + pub fn store_indirect_u16(&mut self, src: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_store_reg(src, Some(base), offset, a64::Size::U16); + } + pub fn store_indirect_u32(&mut self, src: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_store_reg(src, Some(base), offset, a64::Size::U32); + } + pub fn store_indirect_u64(&mut self, src: RawReg, base: RawReg, offset: i32) { + let offset = offset as u32; + self.emit_guest_store_reg(src, Some(base), offset, a64::Size::U64); + } + pub fn store_imm_u8(&mut self, offset: i32, value: i32) { + let offset = offset as u32; + self.emit_guest_store_imm(u64::from(value as u32), None, offset, a64::Size::U8); + } + pub fn store_imm_u16(&mut self, offset: i32, value: i32) { + let offset = offset as u32; + self.emit_guest_store_imm(u64::from(value as u32), None, offset, a64::Size::U16); + } + pub fn store_imm_u32(&mut self, offset: i32, value: i32) { + let offset = offset as u32; + self.emit_guest_store_imm(u64::from(value as u32), None, offset, a64::Size::U32); + } + pub fn store_imm_u64(&mut self, offset: i32, value: i32) { + let offset = offset as u32; + self.emit_guest_store_imm(i64::from(value) as u64, None, offset, a64::Size::U64); + } + pub fn store_imm_indirect_u8(&mut self, base: RawReg, offset: i32, value: i32) { + let offset = offset as u32; + self.emit_guest_store_imm(u64::from(value as u32), Some(base), offset, a64::Size::U8); + } + pub fn store_imm_indirect_u16(&mut self, base: RawReg, offset: i32, value: i32) { + let offset = offset as u32; + self.emit_guest_store_imm(u64::from(value as u32), Some(base), offset, a64::Size::U16); + } + pub fn store_imm_indirect_u32(&mut self, base: RawReg, offset: i32, value: i32) { + let offset = offset as u32; + self.emit_guest_store_imm(u64::from(value as u32), Some(base), offset, a64::Size::U32); + } + pub fn store_imm_indirect_u64(&mut self, base: RawReg, offset: i32, value: i32) { + let offset = offset as u32; + self.emit_guest_store_imm(i64::from(value) as u64, Some(base), offset, a64::Size::U64); + } + pub fn sub_32(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::sub_reg(a64::RegSize::W, conv_reg(d), conv_reg(s1), conv_reg(s2))); + self.finish32(d); + } + pub fn sub_64(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + self.push(a64::sub_reg(a64::RegSize::X, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub(crate) fn trace_execution(&mut self, code_offset: Option) { + // Fixed-length prelude (like the x86 backend) so `step_prelude_length` is exact: the segfault + // resume path skips it by that amount. `code_offset` is materialized with a forced movz+movk. + let step_label = self.step_label; + let origin = self.asm.len(); + let has_offset = code_offset.is_some(); + if let Some(code_offset) = code_offset { + use a64::RegSize::X; + self.push(a64::movz(X, SCRATCH2, code_offset as u16, 0)); + self.push(a64::movk(X, SCRATCH2, (code_offset >> 16) as u16, 1)); + self.vmctx_addr(SCRATCH0, S::offset_table().program_counter); + self.push(a64::str_imm(a64::Size::U32, SCRATCH2, SCRATCH0, 0)); + self.vmctx_addr(SCRATCH0, S::offset_table().next_program_counter); + self.push(a64::str_imm(a64::Size::U32, SCRATCH2, SCRATCH0, 0)); + } + self.push(a64::bl(step_label)); + if has_offset { + debug_assert_eq!(self.asm.len() - origin, step_prelude_length::()); + } + } + pub fn trap(&mut self, code_offset: u32) { + self.st_vmctx_imm32(code_offset, S::offset_table().program_counter); + let trap_label = self.trap_label; + self.push(a64::bl(trap_label)); + } + pub fn xnor(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + // d = ~(s1 ^ s2) = s1 ^ ~s2 + let sz = self.reg_size(); + self.push(a64::eon_reg(sz, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn xor(&mut self, d: RawReg, s1: RawReg, s2: RawReg) { + let sz = self.reg_size(); + self.push(a64::eor_reg(sz, conv_reg(d), conv_reg(s1), conv_reg(s2))); + } + pub fn xor_imm(&mut self, d: RawReg, s1: RawReg, s2: i32) { + let s2 = s2 as u32; + let sz = self.reg_size(); + if let Some(imm) = self.logical_imm(s2) { + self.push(a64::eor_imm(sz, conv_reg(d), conv_reg(s1), imm)); + } else { + let r = self.imm_reg(s2); + self.push(a64::eor_reg(sz, conv_reg(d), conv_reg(s1), r)); + } + } + pub fn zero_extend_16(&mut self, d: RawReg, s: RawReg) { + let sz = self.reg_size(); + self.push(a64::ubfm(sz, conv_reg(d), conv_reg(s), 0, 15)); + } +} + +// ---- Free functions used by the sandbox (parallel to amd64.rs) ---- +// `on_signal_trap`/`on_page_fault` are zygote-only; the generic sandbox recovers from signals inline in generic.rs. + +/// Only here to match the x86 signature; never constructed. +#[cfg(feature = "generic-sandbox")] +#[allow(dead_code)] +#[derive(Copy, Clone)] +pub(crate) enum MemsetKind { + Inline, + Trampoline, +} + +/// Always `None`: the AArch64 `memset` loop keeps its progress in A0/A2 and charges gas per byte, +/// so re-entering the instruction resumes the fill — no fixup or continuation needed. +#[cfg(feature = "generic-sandbox")] +pub(crate) fn are_we_executing_memset(_compiled_module: &crate::compiler::CompiledModule, _offset: u64) -> Option +where + S: Sandbox, +{ + None +} + +/// Reads the 32-bit gas cost embedded in a block's metering stub. +pub fn extract_gas_cost(machine_code: &[u8], basic_block_machine_code_offset: usize) -> u32 +where + S: Sandbox, +{ + let p = basic_block_machine_code_offset + GAS_COST_OFFSET; + let xs = &machine_code[p..p + 4]; + u32::from_le_bytes([xs[0], xs[1], xs[2], xs[3]]) +} + +/// Byte length of a block's gas-metering stub (see `emit_gas_metering_stub`): Sync adds a `cmp`/`b.ge`/`udf`. +pub fn gas_metering_stub_length(kind: GasMeteringKind) -> usize { + match kind { + GasMeteringKind::Sync => GAS_METERING_TRAP_OFFSET as usize + 4, // up to and including the `udf` + GasMeteringKind::Async => 28, // ends after the `str` at +24 + } +} + +#[inline(always)] +pub fn step_prelude_length() -> usize +where + S: Sandbox, +{ + // 7 fixed-length instructions; see `trace_execution`, whose debug_assert verifies this. + 7 * 4 +} diff --git a/crates/polkavm/src/compiler/amd64.rs b/crates/polkavm/src/compiler/amd64.rs index ed0f0ffc3..5e0f40bab 100644 --- a/crates/polkavm/src/compiler/amd64.rs +++ b/crates/polkavm/src/compiler/amd64.rs @@ -584,7 +584,7 @@ where SandboxKind::Generic => { #[cfg(feature = "generic-sandbox")] { - let offset = crate::sandbox::generic::GUEST_MEMORY_TO_VMCTX_OFFSET as i32 + offset as i32; + let offset = crate::sandbox::GUEST_MEMORY_TO_VMCTX_OFFSET as i32 + offset as i32; reg_indirect(RegSize::R64, GENERIC_SANDBOX_MEMORY_REG + offset) } diff --git a/crates/polkavm/src/config.rs b/crates/polkavm/src/config.rs index b4717b8c4..3a806695c 100644 --- a/crates/polkavm/src/config.rs +++ b/crates/polkavm/src/config.rs @@ -55,6 +55,7 @@ impl BackendKind { pub enum SandboxKind { Linux, Generic, + Hypervisor, } impl core::fmt::Display for SandboxKind { @@ -62,6 +63,7 @@ impl core::fmt::Display for SandboxKind { let name = match self { SandboxKind::Linux => "linux", SandboxKind::Generic => "generic", + SandboxKind::Hypervisor => "hypervisor", }; fmt.write_str(name) @@ -77,9 +79,11 @@ impl SandboxKind { Ok(Some(SandboxKind::Linux)) } else if s == "generic" { Ok(Some(SandboxKind::Generic)) + } else if s == "hypervisor" { + Ok(Some(SandboxKind::Hypervisor)) } else { Err(Error::from_static_str( - "invalid value of POLKAVM_SANDBOX; supported values are: 'linux', 'generic'", + "invalid value of POLKAVM_SANDBOX; supported values are: 'linux', 'generic', 'hypervisor'", )) } } @@ -90,8 +94,9 @@ impl SandboxKind { if_compiler_is_supported! { { match self { - SandboxKind::Linux => cfg!(target_os = "linux"), + SandboxKind::Linux => cfg!(all(target_os = "linux", target_arch = "x86_64")), SandboxKind::Generic => cfg!(feature = "generic-sandbox"), + SandboxKind::Hypervisor => cfg!(all(feature = "hypervisor-sandbox", target_os = "macos", target_arch = "aarch64")), } } else { false diff --git a/crates/polkavm/src/error.rs b/crates/polkavm/src/error.rs index c2d4b1ba0..0d5c3de6c 100644 --- a/crates/polkavm/src/error.rs +++ b/crates/polkavm/src/error.rs @@ -42,7 +42,7 @@ impl From for Error { } if_compiler_is_supported! { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] impl From for Error { #[cold] fn from(error: polkavm_linux_raw::Error) -> Self { @@ -60,6 +60,14 @@ if_compiler_is_supported! { Self(ErrorKind::Owned(error.to_string())) } } + + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + impl From for Error { + #[cold] + fn from(error: crate::sandbox::hypervisor::Error) -> Self { + Self(ErrorKind::Owned(error.to_string())) + } + } } impl Error { diff --git a/crates/polkavm/src/lib.rs b/crates/polkavm/src/lib.rs index 0f3d92714..ac4a6fdc4 100644 --- a/crates/polkavm/src/lib.rs +++ b/crates/polkavm/src/lib.rs @@ -10,12 +10,21 @@ #[cfg(all( not(miri), - target_arch = "x86_64", + feature = "std", any( - target_os = "linux", - all(feature = "generic-sandbox", any(target_os = "macos", target_os = "freebsd")) + all( + target_arch = "x86_64", + any( + target_os = "linux", + all(feature = "generic-sandbox", any(target_os = "macos", target_os = "freebsd")) + ), + ), + all( + target_arch = "aarch64", + any(feature = "generic-sandbox", feature = "hypervisor-sandbox"), + any(target_os = "macos", target_os = "linux") + ), ), - feature = "std", ))] macro_rules! if_compiler_is_supported { ({ @@ -33,12 +42,21 @@ macro_rules! if_compiler_is_supported { #[cfg(not(all( not(miri), - target_arch = "x86_64", + feature = "std", any( - target_os = "linux", - all(feature = "generic-sandbox", any(target_os = "macos", target_os = "freebsd")) + all( + target_arch = "x86_64", + any( + target_os = "linux", + all(feature = "generic-sandbox", any(target_os = "macos", target_os = "freebsd")) + ), + ), + all( + target_arch = "aarch64", + any(feature = "generic-sandbox", feature = "hypervisor-sandbox"), + any(target_os = "macos", target_os = "linux") + ), ), - feature = "std", )))] macro_rules! if_compiler_is_supported { ({ @@ -93,16 +111,18 @@ mod module_cache; if_compiler_is_supported! { mod compiler; + // Page tracking for the generic sandbox; the hypervisor tracks its own guest page table. + #[cfg_attr(not(feature = "generic-sandbox"), allow(dead_code))] mod page_set; mod sandbox; - #[cfg(all(target_os = "linux", not(feature = "export-internals-for-testing")))] + #[cfg(all(target_os = "linux", target_arch = "x86_64", not(feature = "export-internals-for-testing")))] mod generic_allocator; - #[cfg(all(target_os = "linux", not(feature = "export-internals-for-testing")))] + #[cfg(all(target_os = "linux", target_arch = "x86_64", not(feature = "export-internals-for-testing")))] mod bit_mask; - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] mod shm_allocator; } diff --git a/crates/polkavm/src/sandbox.rs b/crates/polkavm/src/sandbox.rs index 9a96ba3e3..7d4fdb477 100644 --- a/crates/polkavm/src/sandbox.rs +++ b/crates/polkavm/src/sandbox.rs @@ -25,7 +25,10 @@ macro_rules! get_field_offset { #[cfg(feature = "generic-sandbox")] pub mod generic; -#[cfg(target_os = "linux")] +#[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] +pub mod hypervisor; + +#[cfg(all(target_os = "linux", target_arch = "x86_64"))] pub mod linux; // This is literally the only thing we need from `libc` on Linux, so instead of including @@ -74,15 +77,23 @@ pub trait SandboxProgram: Clone { fn machine_code(&self) -> &[u8]; } +/// Where the VM context sits relative to the guest memory base. Both the generic and the hypervisor +/// sandbox place it one page below, which is what lets them share the AArch64 codegen. +#[cfg(any(target_os = "linux", feature = "generic-sandbox", feature = "hypervisor-sandbox"))] +pub(crate) const GUEST_MEMORY_TO_VMCTX_OFFSET: isize = -4096; + pub struct OffsetTable { pub arg: usize, pub gas: usize, pub heap_info: usize, pub next_native_program_counter: usize, + // Only read by the x86 `memset` codegen; AArch64's loop tracks progress in A0/A2 instead. + #[cfg_attr(target_arch = "aarch64", allow(dead_code))] pub memset_continuation: usize, pub next_program_counter: usize, pub program_counter: usize, pub regs: usize, + #[allow(dead_code)] // Used by the Linux zygote sandbox; unused by the generic sandbox. pub futex: usize, } @@ -112,6 +123,7 @@ pub(crate) trait Sandbox: Sized { fn recycle(sandbox: Box, global: &Self::GlobalState) -> Result<(), Self::Error>; fn address_table() -> AddressTable; fn offset_table() -> OffsetTable; + #[allow(dead_code)] // Used by the Linux zygote sandbox tooling; unused by the generic sandbox. fn idle_worker_pids(global: &Self::GlobalState) -> Vec; fn run(&mut self) -> Result; @@ -220,17 +232,19 @@ where } pub(crate) enum GlobalStateKind { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] Linux(crate::sandbox::linux::GlobalState), #[cfg(feature = "generic-sandbox")] Generic(crate::sandbox::generic::GlobalState), + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + Hypervisor(crate::sandbox::hypervisor::GlobalState), } impl GlobalStateKind { pub(crate) fn new(kind: SandboxKind, config: &Config) -> Result { match kind { SandboxKind::Linux => { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] { Ok(Self::Linux( crate::sandbox::linux::GlobalState::new(config) @@ -238,7 +252,7 @@ impl GlobalStateKind { )) } - #[cfg(not(target_os = "linux"))] + #[cfg(not(all(target_os = "linux", target_arch = "x86_64")))] { unreachable!() } @@ -257,13 +271,27 @@ impl GlobalStateKind { unreachable!() } } + SandboxKind::Hypervisor => { + #[cfg(all(feature = "hypervisor-sandbox", target_arch = "aarch64"))] + { + Ok(Self::Hypervisor( + crate::sandbox::hypervisor::GlobalState::new(config) + .map_err(|error| format!("failed to initialize hypervisor sandbox: {error}"))?, + )) + } + + #[cfg(not(all(feature = "hypervisor-sandbox", target_arch = "aarch64")))] + { + unreachable!() + } + } } } pub(crate) fn idle_worker_pids(&self) -> Vec { #[allow(unreachable_patterns)] match self { - #[cfg(target_os = "linux")] + #[cfg(all(target_os = "linux", target_arch = "x86_64"))] GlobalStateKind::Linux(state) => crate::sandbox::linux::Sandbox::idle_worker_pids(state), _ => Vec::new(), } @@ -297,7 +325,7 @@ where } // This is the same for both sandboxes. -#[cfg(any(target_os = "linux", feature = "generic-sandbox"))] +#[cfg(any(target_os = "linux", any(feature = "generic-sandbox", feature = "hypervisor-sandbox")))] pub(crate) fn charge_gas_on_entry( module: &Module, pc: ProgramCounter, diff --git a/crates/polkavm/src/sandbox/generic.rs b/crates/polkavm/src/sandbox/generic.rs index 5df2b636f..cb22104d5 100644 --- a/crates/polkavm/src/sandbox/generic.rs +++ b/crates/polkavm/src/sandbox/generic.rs @@ -59,8 +59,9 @@ fn to_usize(value: T) -> Cast { Cast(value, core::marker::PhantomData) } -// On Linux don't depend on the `libc` crate to lower the number of dependencies. -#[cfg(target_os = "linux")] +// On x86_64 Linux don't depend on the `libc` crate to lower the number of dependencies. +// (polkavm-linux-raw is x86_64-only, so aarch64 Linux uses libc — see the `use libc as sys` below.) +#[cfg(all(target_os = "linux", target_arch = "x86_64"))] #[allow(non_camel_case_types)] mod sys { pub use polkavm_linux_raw::{c_int, c_void, siginfo_t, size_t, ucontext as ucontext_t, SIG_DFL, SIG_IGN}; @@ -111,16 +112,45 @@ mod sys { } } -#[cfg(not(target_os = "linux"))] +#[cfg(not(all(target_os = "linux", target_arch = "x86_64")))] use libc as sys; +// Apple Silicon W^X: MAP_JIT regions toggle writability per-thread via pthread_jit_write_protect_np, +// and the icache must be flushed (sys_icache_invalidate) after writing code. +#[cfg(all(target_os = "macos", target_arch = "aarch64"))] +mod jit_mem { + use super::{c_int, c_void, size_t}; + + extern "C" { + pub fn sys_icache_invalidate(start: *mut c_void, len: size_t); + } + + #[inline] + pub unsafe fn set_writable(writable: bool) { + // 1 = executable (write-protected), 0 = writable. + libc::pthread_jit_write_protect_np(c_int::from(!writable)); + } + + #[inline] + pub unsafe fn flush_icache(ptr: *mut c_void, len: size_t) { + sys_icache_invalidate(ptr, len); + } +} + use core::ffi::c_void; use sys::{c_int, size_t, MADV_DONTNEED, MADV_FREE, MAP_ANONYMOUS, MAP_FIXED, MAP_PRIVATE, PROT_EXEC, PROT_READ, PROT_WRITE}; -pub(crate) const GUEST_MEMORY_TO_VMCTX_OFFSET: isize = -4096; +use crate::sandbox::GUEST_MEMORY_TO_VMCTX_OFFSET; +// These must match the gas-metering stub layout emitted by the compiler backend for this target. +#[cfg(target_arch = "x86_64")] const GAS_METERING_TRAP_OFFSET: u64 = 3; +#[cfg(target_arch = "x86_64")] const GAS_COST_GENERIC_SANDBOX_OFFSET: usize = 7; +#[cfg(target_arch = "aarch64")] +const GAS_METERING_TRAP_OFFSET: u64 = 36; +#[cfg(target_arch = "aarch64")] +const GAS_COST_GENERIC_SANDBOX_OFFSET: usize = 8; fn get_guest_memory_offset() -> usize { get_native_page_size() @@ -150,6 +180,10 @@ impl From for Error { pub struct Mmap { pointer: *mut c_void, length: usize, + // MAP_JIT region (Apple Silicon): uses per-thread write-protect toggling instead of mprotect. + // Only read on macOS-aarch64; kept as a field on all targets for a uniform constructor. + #[allow(dead_code)] + is_jit: bool, } // SAFETY: The ownership of an mmapped piece of memory can be safely transferred to other threads. @@ -168,7 +202,33 @@ impl Mmap { pointer }; - Ok(Self { pointer, length }) + Ok(Self { + pointer, + length, + is_jit: false, + }) + } + + /// Like [`Mmap::reserve_address_space`], but uses `MAP_JIT` on Apple Silicon for executable code. + pub fn reserve_jit_address_space(length: size_t) -> Result { + #[cfg(all(target_os = "macos", target_arch = "aarch64"))] + { + // SAFETY: `MAP_FIXED` is not specified, so this is always safe. + let mut map = unsafe { + Mmap::raw_mmap( + core::ptr::null_mut(), + length, + PROT_READ | PROT_WRITE | PROT_EXEC, + MAP_ANONYMOUS | MAP_PRIVATE | sys::MAP_JIT, + )? + }; + map.is_jit = true; + Ok(map) + } + #[cfg(not(all(target_os = "macos", target_arch = "aarch64")))] + { + Mmap::reserve_address_space(length) + } } fn mmap_within(&mut self, offset: usize, length: usize, protection: c_int) -> Result<(), Error> { @@ -252,11 +312,36 @@ impl Mmap { protection: c_int, callback: impl FnOnce(&mut [u8]), ) -> Result<(), Error> { + // MAP_JIT regions toggle per-thread write protection + flush the icache instead of mprotect. + #[cfg(all(target_os = "macos", target_arch = "aarch64"))] + if self.is_jit { + if !offset.checked_add(length).map_or(false, |end| end <= self.length) { + return Err("out of bounds modify_and_protect".into()); + } + + // SAFETY: We're toggling write protection around the write of code we own. + unsafe { + jit_mem::set_writable(true); + callback(&mut self.as_slice_mut()[offset..offset + length]); + jit_mem::set_writable(false); + let ptr = self.pointer.cast::().add(offset).cast::(); + jit_mem::flush_icache(ptr, length); + } + return Ok(()); + } + self.mprotect(offset, length, PROT_READ | PROT_WRITE)?; callback(&mut self.as_slice_mut()[offset..offset + length]); if protection != PROT_READ | PROT_WRITE { self.mprotect(offset, length, protection)?; } + // AArch64 has separate I/D caches: after writing code we must flush the I-cache so the CPU + // doesn't execute stale instructions. (macOS handles this in the MAP_JIT path above.) + #[cfg(all(target_arch = "aarch64", not(target_os = "macos")))] + unsafe { + let start = self.pointer.cast::().add(offset).cast::(); + clear_instruction_cache(start, start.add(length)); + } Ok(()) } @@ -298,6 +383,7 @@ impl Default for Mmap { Self { pointer: core::ptr::NonNull::::dangling().as_ptr().cast::(), length: 0, + is_jit: false, } } } @@ -308,186 +394,256 @@ impl Drop for Mmap { } } +// AArch64 I-cache maintenance for freshly written code (non-macOS; macOS uses sys_icache_invalidate). +#[cfg(all(target_arch = "aarch64", not(target_os = "macos")))] +#[inline] +unsafe fn clear_instruction_cache(start: *mut core::ffi::c_char, end: *mut core::ffi::c_char) { + extern "C" { + // Provided by the compiler runtime (compiler-builtins / libgcc); performs the + // architecture-appropriate I-cache maintenance over the given range. + fn __clear_cache(start: *mut core::ffi::c_char, end: *mut core::ffi::c_char); + } + __clear_cache(start, end); +} + static mut OLD_SIGSEGV: sys::sigaction = unsafe { core::mem::zeroed() }; static mut OLD_SIGILL: sys::sigaction = unsafe { core::mem::zeroed() }; -#[cfg(any(target_os = "macos", target_os = "freebsd"))] +#[cfg(any(target_os = "macos", target_os = "freebsd", all(target_os = "linux", target_arch = "aarch64")))] static mut OLD_SIGBUS: sys::sigaction = unsafe { core::mem::zeroed() }; unsafe extern "C" fn signal_handler(signal: c_int, info: &sys::siginfo_t, context: &sys::ucontext_t) { let old = match signal { sys::SIGSEGV => core::ptr::addr_of!(OLD_SIGSEGV), sys::SIGILL => core::ptr::addr_of!(OLD_SIGILL), - #[cfg(any(target_os = "macos", target_os = "freebsd"))] + #[cfg(any(target_os = "macos", target_os = "freebsd", all(target_os = "linux", target_arch = "aarch64")))] sys::SIGBUS => core::ptr::addr_of!(OLD_SIGBUS), _ => unreachable!("received unknown signal"), }; let vmctx = THREAD_VMCTX.with(|thread_ctx| *thread_ctx.get()); if !vmctx.is_null() { - #[cfg(target_os = "macos")] - macro_rules! macos_reg_field { - (rax) => { - (*context.uc_mcontext).__ss.__rax - }; - (rbx) => { - (*context.uc_mcontext).__ss.__rbx - }; - (rcx) => { - (*context.uc_mcontext).__ss.__rcx - }; - (rdx) => { - (*context.uc_mcontext).__ss.__rdx - }; - (rdi) => { - (*context.uc_mcontext).__ss.__rdi - }; - (rsi) => { - (*context.uc_mcontext).__ss.__rsi - }; - (rbp) => { - (*context.uc_mcontext).__ss.__rbp - }; - (rsp) => { - (*context.uc_mcontext).__ss.__rsp - }; - (r8) => { - (*context.uc_mcontext).__ss.__r8 - }; - (r9) => { - (*context.uc_mcontext).__ss.__r9 - }; - (r10) => { - (*context.uc_mcontext).__ss.__r10 - }; - (r11) => { - (*context.uc_mcontext).__ss.__r11 - }; - (r12) => { - (*context.uc_mcontext).__ss.__r12 - }; - (r13) => { - (*context.uc_mcontext).__ss.__r13 - }; - (r14) => { - (*context.uc_mcontext).__ss.__r14 - }; - (r15) => { - (*context.uc_mcontext).__ss.__r15 + #[cfg(target_arch = "aarch64")] + { + // SAFETY: On a fault inside guest code the kernel hands us a valid thread state. + // macOS and Linux expose the AArch64 register file differently in the ucontext: + // macOS via `uc_mcontext->__ss` (a pointer), Linux via an inline `uc_mcontext`. + #[cfg(target_os = "macos")] + let (pc, regs) = { + let ss = &(*context.uc_mcontext).__ss; + (ss.__pc, ss.__x) }; - (rip) => { - (*context.uc_mcontext).__ss.__rip + #[cfg(target_os = "linux")] + let (pc, regs) = { + let mc = &context.uc_mcontext; + (mc.pc, mc.regs) }; + let vmctx = &mut *vmctx; + + use polkavm_common::regmap::{to_native_reg, NativeReg, TMP_REG}; + // Guest registers and TMP_REG live in x0..x13, so we can index the register file directly. + for reg in polkavm_common::program::Reg::ALL { + let idx = to_native_reg(reg).idx() as usize; + vmctx.regs[reg as usize] = regs[idx]; + } + polkavm_common::static_assert!(TMP_REG.equals(NativeReg::X13)); + vmctx.tmp_reg.store(regs[TMP_REG.idx() as usize], Ordering::Relaxed); + + if vmctx.program_range.contains(&pc) { + vmctx.next_native_program_counter.store(pc, Ordering::Relaxed); + // No `trapno` on AArch64: treat in-range SIGSEGV/SIGBUS as a page fault (gas UDF is SIGILL). + let is_page_fault = signal == sys::SIGSEGV || signal == sys::SIGBUS; + if is_page_fault { + // macOS reads the fault address from siginfo; Linux AArch64 carries it in the + // sigcontext (si_addr is a method, not a field, in libc there). + #[cfg(target_os = "macos")] + let fault_address = info.si_addr as u64; + #[cfg(target_os = "linux")] + let fault_address = { + let _ = info; + context.uc_mcontext.fault_address + }; + log::trace!("Page fault at 0x{fault_address:x} (pc: 0x{pc:x})"); + trigger_exit(vmctx, ExitReason::Segfault(fault_address)); + } else { + log::trace!("Signal received at 0x{pc:x}"); + trigger_exit(vmctx, ExitReason::Signal); + } + } else { + // Out-of-range pc during guest execution: a `br` to an invalid jump-table entry (sentinel + // or zeroed slot) jumped there and faulted. jump_indirect stashed the site in + // next_native_program_counter, so report a trap there. + log::trace!("Invalid jump to 0x{pc:x}"); + trigger_exit(vmctx, ExitReason::Signal); + } } - macro_rules! fetch_reg { - ($reg:ident) => {{ + #[cfg(target_arch = "x86_64")] + { + #[cfg(target_os = "macos")] + macro_rules! macos_reg_field { + (rax) => { + (*context.uc_mcontext).__ss.__rax + }; + (rbx) => { + (*context.uc_mcontext).__ss.__rbx + }; + (rcx) => { + (*context.uc_mcontext).__ss.__rcx + }; + (rdx) => { + (*context.uc_mcontext).__ss.__rdx + }; + (rdi) => { + (*context.uc_mcontext).__ss.__rdi + }; + (rsi) => { + (*context.uc_mcontext).__ss.__rsi + }; + (rbp) => { + (*context.uc_mcontext).__ss.__rbp + }; + (rsp) => { + (*context.uc_mcontext).__ss.__rsp + }; + (r8) => { + (*context.uc_mcontext).__ss.__r8 + }; + (r9) => { + (*context.uc_mcontext).__ss.__r9 + }; + (r10) => { + (*context.uc_mcontext).__ss.__r10 + }; + (r11) => { + (*context.uc_mcontext).__ss.__r11 + }; + (r12) => { + (*context.uc_mcontext).__ss.__r12 + }; + (r13) => { + (*context.uc_mcontext).__ss.__r13 + }; + (r14) => { + (*context.uc_mcontext).__ss.__r14 + }; + (r15) => { + (*context.uc_mcontext).__ss.__r15 + }; + (rip) => { + (*context.uc_mcontext).__ss.__rip + }; + } + + macro_rules! fetch_reg { + ($reg:ident) => {{ + #[cfg(target_os = "linux")] + { + context.uc_mcontext.$reg as u64 + } + #[cfg(target_os = "macos")] + { + macos_reg_field!($reg) as u64 + } + #[cfg(target_os = "freebsd")] + { + context.uc_mcontext.mc_($reg) as u64 + } + }}; + } + + const X86_TRAP_PF: u64 = 14; + let is_page_fault = { #[cfg(target_os = "linux")] { - context.uc_mcontext.$reg as u64 + signal == sys::SIGSEGV && context.uc_mcontext.trapno == X86_TRAP_PF } #[cfg(target_os = "macos")] { - macos_reg_field!($reg) as u64 + signal == sys::SIGBUS && (*context.uc_mcontext).__es.__trapno as u64 == X86_TRAP_PF } #[cfg(target_os = "freebsd")] { - context.uc_mcontext.mc_($reg) as u64 + signal == sys::SIGBUS && context.uc_mcontext.mc_trapno == X86_TRAP_PF } - }}; - } + }; - const X86_TRAP_PF: u64 = 14; - let is_page_fault = { - #[cfg(target_os = "linux")] - { - signal == sys::SIGSEGV && context.uc_mcontext.trapno == X86_TRAP_PF - } + let rip = fetch_reg!(rip); + let vmctx = &mut *vmctx; + + // On Rosetta 2, the JMP emulation logic doesn't work same as on x64. + // Instead of triggering a GPF immdiately, it jumps to that address and then trigger a PF. + // Therefore the original program counter is lost. + // We fix this problem by storing the program counter in vmctx before jumping. + // See jump_indirect_impl for more details. #[cfg(target_os = "macos")] { - signal == sys::SIGBUS && (*context.uc_mcontext).__es.__trapno as u64 == X86_TRAP_PF - } - #[cfg(target_os = "freebsd")] - { - signal == sys::SIGBUS && context.uc_mcontext.mc_trapno == X86_TRAP_PF - } - }; - - let rip = fetch_reg!(rip); - let vmctx = &mut *vmctx; - - // On Rosetta 2, the JMP emulation logic doesn't work same as on x64. - // Instead of triggering a GPF immdiately, it jumps to that address and then trigger a PF. - // Therefore the original program counter is lost. - // We fix this problem by storing the program counter in vmctx before jumping. - // See jump_indirect_impl for more details. - #[cfg(target_os = "macos")] - { - let is_invalid_rip = (rip >> 48) != 0; - if is_invalid_rip { - log::trace!("Jump table invalid address hit, returning to host"); - trigger_exit(vmctx, ExitReason::Signal); + let is_invalid_rip = (rip >> 48) != 0; + if is_invalid_rip { + log::trace!("Jump table invalid address hit, returning to host"); + trigger_exit(vmctx, ExitReason::Signal); + } } - } - if vmctx.program_range.contains(&rip) { - use polkavm_common::regmap::NativeReg; - for reg in polkavm_common::program::Reg::ALL { - let value = match polkavm_common::regmap::to_native_reg(reg) { - NativeReg::rax => fetch_reg!(rax), - NativeReg::rcx => fetch_reg!(rcx), - NativeReg::rdx => fetch_reg!(rdx), - NativeReg::rbx => fetch_reg!(rbx), - NativeReg::rbp => fetch_reg!(rbp), - NativeReg::rsi => fetch_reg!(rsi), - NativeReg::rdi => fetch_reg!(rdi), - NativeReg::r8 => fetch_reg!(r8), - NativeReg::r9 => fetch_reg!(r9), - NativeReg::r10 => fetch_reg!(r10), - NativeReg::r11 => fetch_reg!(r11), - NativeReg::r12 => fetch_reg!(r12), - NativeReg::r13 => fetch_reg!(r13), - NativeReg::r14 => fetch_reg!(r14), - NativeReg::r15 => fetch_reg!(r15), - }; - vmctx.regs[reg as usize] = value; - } + if vmctx.program_range.contains(&rip) { + use polkavm_common::regmap::NativeReg; + for reg in polkavm_common::program::Reg::ALL { + let value = match polkavm_common::regmap::to_native_reg(reg) { + NativeReg::rax => fetch_reg!(rax), + NativeReg::rcx => fetch_reg!(rcx), + NativeReg::rdx => fetch_reg!(rdx), + NativeReg::rbx => fetch_reg!(rbx), + NativeReg::rbp => fetch_reg!(rbp), + NativeReg::rsi => fetch_reg!(rsi), + NativeReg::rdi => fetch_reg!(rdi), + NativeReg::r8 => fetch_reg!(r8), + NativeReg::r9 => fetch_reg!(r9), + NativeReg::r10 => fetch_reg!(r10), + NativeReg::r11 => fetch_reg!(r11), + NativeReg::r12 => fetch_reg!(r12), + NativeReg::r13 => fetch_reg!(r13), + NativeReg::r14 => fetch_reg!(r14), + NativeReg::r15 => fetch_reg!(r15), + }; + vmctx.regs[reg as usize] = value; + } - // - // RCX is TMP_REG, which is not mapped to any guest register. - // therefore we store it separately here. - // - // N.B Even though it's a volatile register, we still need to - // restore it to handle page faults correctly. - // - polkavm_common::static_assert!(polkavm_common::regmap::TMP_REG.equals(NativeReg::rcx)); - vmctx.tmp_reg.store(fetch_reg!(rcx), Ordering::Relaxed); + // + // RCX is TMP_REG, which is not mapped to any guest register. + // therefore we store it separately here. + // + // N.B Even though it's a volatile register, we still need to + // restore it to handle page faults correctly. + // + polkavm_common::static_assert!(polkavm_common::regmap::TMP_REG.equals(NativeReg::rcx)); + vmctx.tmp_reg.store(fetch_reg!(rcx), Ordering::Relaxed); - vmctx.next_native_program_counter.store(rip, Ordering::Relaxed); + vmctx.next_native_program_counter.store(rip, Ordering::Relaxed); - if is_page_fault { - let fault_address = { - #[cfg(target_os = "linux")] - { - info.__bindgen_anon_1.__bindgen_anon_1._sifields._sigfault._addr as u64 - } - #[cfg(target_os = "macos")] - { - info.si_addr as u64 - } - #[cfg(target_os = "freebsd")] - { - info.si_addr as u64 - } - }; + if is_page_fault { + let fault_address = { + #[cfg(target_os = "linux")] + { + info.__bindgen_anon_1.__bindgen_anon_1._sifields._sigfault._addr as u64 + } + #[cfg(target_os = "macos")] + { + info.si_addr as u64 + } + #[cfg(target_os = "freebsd")] + { + info.si_addr as u64 + } + }; - log::trace!("Page fault at 0x{fault_address:x} (rip: 0x{rip:x})"); - trigger_exit(vmctx, ExitReason::Segfault(fault_address)); - } else { - log::trace!("Signal received at 0x{rip:x}"); - trigger_exit(vmctx, ExitReason::Signal); + log::trace!("Page fault at 0x{fault_address:x} (rip: 0x{rip:x})"); + trigger_exit(vmctx, ExitReason::Segfault(fault_address)); + } else { + log::trace!("Signal received at 0x{rip:x}"); + trigger_exit(vmctx, ExitReason::Signal); + } } - } + } // end #[cfg(target_arch = "x86_64")] } // This signal is unrelated to anything the guest program did; proceed normally. @@ -524,7 +680,7 @@ unsafe fn register_signal_handler_for_signal(signal: c_int, old_sa: *mut sys::si unsafe fn register_signal_handlers() -> Result<(), Error> { register_signal_handler_for_signal(sys::SIGSEGV, core::ptr::addr_of_mut!(OLD_SIGSEGV))?; register_signal_handler_for_signal(sys::SIGILL, core::ptr::addr_of_mut!(OLD_SIGILL))?; - #[cfg(any(target_os = "macos", target_os = "freebsd"))] + #[cfg(any(target_os = "macos", target_os = "freebsd", all(target_os = "linux", target_arch = "aarch64")))] register_signal_handler_for_signal(sys::SIGBUS, core::ptr::addr_of_mut!(OLD_SIGBUS))?; Ok(()) } @@ -572,6 +728,7 @@ unsafe fn sysreturn(vmctx: &mut VmCtx) -> ! { debug_assert_ne!(vmctx.return_stack_pointer, 0); // SAFETY: This function can only be called while we're executing guest code. + #[cfg(target_arch = "x86_64")] unsafe { core::arch::asm!(r#" // Restore the stack pointer to its original value. @@ -584,6 +741,23 @@ unsafe fn sysreturn(vmctx: &mut VmCtx) -> ! { options(noreturn) ); } + + // SAFETY: This function can only be called while we're executing guest code. + #[cfg(target_arch = "aarch64")] + unsafe { + core::arch::asm!(r#" + // Restore the stack pointer to its original value. + ldr x17, [{vmctx}, #8] + mov sp, x17 + + // Jump back to the saved return address. + ldr x17, [{vmctx}] + br x17 + "#, + vmctx = in(reg) vmctx, + options(noreturn) + ); + } } #[repr(C)] @@ -1096,7 +1270,7 @@ impl Sandbox { self.vmctx() .next_native_program_counter - .store(native_code_origin + offset as u64, Ordering::Relaxed); + .store(native_code_origin + offset, Ordering::Relaxed); let offset = offset as usize + GAS_COST_GENERIC_SANDBOX_OFFSET; let Some(gas_cost) = &compiled_module.machine_code().get(offset..offset + 4) else { @@ -1166,6 +1340,14 @@ impl Sandbox { )); }; + // Re-entry point: mid-`memset` the recorded continuation, else the faulting instruction's + // start (recomputed on AArch64). Resolved here while `compiled_module` is still borrowed. + let restart_address = if memset.is_some() { + memset_continuation + } else { + compiled_module.resume_native_address_for_pagefault(program_counter, machine_code_address, self.gas_metering) + }; + self.vmctx().program_counter.store(program_counter.0, Ordering::Relaxed); self.vmctx().next_program_counter.store(program_counter.0, Ordering::Relaxed); self.is_program_counter_valid = true; @@ -1187,11 +1369,6 @@ impl Sandbox { } if let Some(is_write_protected) = segfault_kind { - let restart_address = if memset.is_some() { - memset_continuation - } else { - machine_code_address - }; self.vmctx().next_native_program_counter.store(restart_address, Ordering::Relaxed); Ok(InterruptKind::Segfault(Segfault { @@ -1263,19 +1440,16 @@ impl Sandbox { fn madvise_remove(&mut self, address: u32, length: u32) -> Result<(), Error> { assert!(self.dynamic_paging_enabled); - self.memory.madvise( - self.guest_memory_offset + cast(address).to_usize(), - cast(length).to_usize(), - MADV_DONTNEED, - )?; - - self.memory.madvise( - self.guest_memory_offset + cast(address).to_usize(), - cast(length).to_usize(), - MADV_FREE, - )?; - - Ok(()) + let offset = self.guest_memory_offset + cast(address).to_usize(); + let length = cast(length).to_usize(); + if cfg!(target_os = "macos") { + // macOS MADV_DONTNEED/MADV_FREE don't zero-fill private anon pages like Linux does, so + // remap fresh zero pages to actually clear the region (dev host only). + self.memory.mmap_within(offset, length, PROT_READ | PROT_WRITE) + } else { + self.memory.madvise(offset, length, MADV_DONTNEED)?; + self.memory.madvise(offset, length, MADV_FREE) + } } } @@ -1300,6 +1474,8 @@ impl super::Sandbox for Sandbox { } fn downcast_module(module: &Module) -> &CompiledModule { + // The `_` arm is for the other sandbox kinds (Linux); on a generic-only build it's unreachable. + #[allow(unreachable_patterns, clippy::match_wildcard_for_single_variants)] match module.compiled_module() { CompiledModuleKind::Generic(ref module) => module, _ => unreachable!(), @@ -1307,7 +1483,7 @@ impl super::Sandbox for Sandbox { } fn downcast_global_state(global: &crate::sandbox::GlobalStateKind) -> &Self::GlobalState { - #[allow(clippy::match_wildcard_for_single_variants)] + #[allow(unreachable_patterns, clippy::match_wildcard_for_single_variants)] match global { crate::sandbox::GlobalStateKind::Generic(global) => global, _ => unreachable!(), @@ -1322,7 +1498,8 @@ impl super::Sandbox for Sandbox { } fn reserve_address_space() -> Result { - Mmap::reserve_address_space(VM_SANDBOX_MAXIMUM_NATIVE_CODE_SIZE as usize + VM_SANDBOX_MAXIMUM_JUMP_TABLE_VIRTUAL_SIZE as usize) + // This region holds executable JIT code, so on Apple Silicon it must be `MAP_JIT`. + Mmap::reserve_jit_address_space(VM_SANDBOX_MAXIMUM_NATIVE_CODE_SIZE as usize + VM_SANDBOX_MAXIMUM_JUMP_TABLE_VIRTUAL_SIZE as usize) } fn prepare_program( @@ -1375,11 +1552,12 @@ impl super::Sandbox for Sandbox { let mut memory_map = Vec::new(); if cfg.ro_data_size() > 0 { let mut ro_data = init.guest_init.ro_data.to_vec(); - let physical_size = align_to_next_page_usize(native_page_size, ro_data.len()).unwrap(); - ro_data.resize(physical_size, 0); - let physical_size = physical_size.try_into().expect("overflow"); - let virtual_size = cfg.ro_data_size(); + // Clamp to the guest region: a 16K native page can exceed it, underflowing `padding` below. + let physical_size: u32 = u32::try_from(align_to_next_page_usize(native_page_size, ro_data.len()).unwrap()) + .expect("overflow") + .min(virtual_size); + ro_data.resize(physical_size as usize, 0); if physical_size > 0 { memory_map.push(ProgramMap { address: cfg.ro_data_address(), @@ -1402,11 +1580,12 @@ impl super::Sandbox for Sandbox { if cfg.rw_data_size() > 0 { let mut rw_data = init.guest_init.rw_data.to_vec(); - let physical_size = align_to_next_page_usize(native_page_size, rw_data.len()).unwrap(); - rw_data.resize(physical_size, 0); - let physical_size = physical_size.try_into().expect("overflow"); - let virtual_size = cfg.rw_data_size(); + // See the ro_data clamp. + let physical_size: u32 = u32::try_from(align_to_next_page_usize(native_page_size, rw_data.len()).unwrap()) + .expect("overflow") + .min(virtual_size); + rw_data.resize(physical_size as usize, 0); if physical_size > 0 { memory_map.push(ProgramMap { address: cfg.rw_data_address(), @@ -1514,6 +1693,18 @@ impl super::Sandbox for Sandbox { return Err(Error::from("module already loaded")); } + // Guest page protections are enforced with `mprotect`, so their granularity is the host page + // size. A module with smaller pages is under-protected: an out-of-bounds access only faults + // once it crosses the next *host* page boundary. Nothing here can fix that, so say so. + if get_native_page_size() > module.memory_map().page_size() as usize { + log::warn!( + "Module page size ({}) is smaller than the native page size ({}); \ + out-of-bounds guest accesses will not be caught until the next native page boundary", + module.memory_map().page_size(), + get_native_page_size(), + ); + } + if module.is_dynamic_paging() && get_native_page_size() != module.memory_map().page_size() as usize { return Err(Error::from( "dynamic paging is currently unsupported if the module's page size doesn't match the native page size", @@ -1663,7 +1854,7 @@ impl super::Sandbox for Sandbox { let entry_point = compiled_module.sandbox_program.0.sysenter_address; log::trace!("Jumping into guest program: 0x{:x}", entry_point); - self.set_aux_data_permission_for_guest().map_err(Error::from)?; + self.set_aux_data_permission_for_guest()?; #[allow(clippy::undocumented_unsafe_blocks)] unsafe { @@ -1673,6 +1864,7 @@ impl super::Sandbox for Sandbox { let guest_memory = self.memory.as_ptr().cast::().add(self.guest_memory_offset); let tmp_reg = self.vmctx().tmp_reg.load(Ordering::Relaxed); + #[cfg(target_arch = "x86_64")] core::arch::asm!(r#" push rbp push rbx @@ -1717,11 +1909,50 @@ impl super::Sandbox for Sandbox { in("r13") guest_memory, ); + #[cfg(target_arch = "aarch64")] + core::arch::asm!(r#" + // Save callee-saved regs (x19..x30): the guest path re-enters at `2:` via sysreturn's + // branch, and the `-> !` syscall trampolines clobber them without running an epilogue. + // LLVM forbids x19 as an asm operand/clobber, so preserve them manually (like x86's + // push rbp/rbx). The return sp is saved *after* the pushes so sysreturn pops correctly. + stp x19, x20, [sp, #-96]! + stp x21, x22, [sp, #16] + stp x23, x24, [sp, #32] + stp x25, x26, [sp, #48] + stp x27, x28, [sp, #64] + stp x29, x30, [sp, #80] + + adr x17, 2f + str x17, [{vmctx}] + mov x17, sp + str x17, [{vmctx}, #8] + + // sysenter loads the guest registers from the vmctx via x14/AUX_TMP_REG (memory base). + br {entry_point} + + // We re-enter here on exit (via `sysreturn`). + 2: + ldp x21, x22, [sp, #16] + ldp x23, x24, [sp, #32] + ldp x25, x26, [sp, #48] + ldp x27, x28, [sp, #64] + ldp x29, x30, [sp, #80] + ldp x19, x20, [sp], #96 + "#, + entry_point = in(reg) entry_point, + vmctx = in(reg) vmctx, + // x14 = AUX_TMP_REG holds the guest memory base; x13 = TMP_REG. + in("x14") guest_memory, + in("x13") tmp_reg, + // Caller-saved regs (x0..x17) are clobbered by the guest; callee-saved are preserved above. + clobber_abi("C"), + ); + THREAD_VMCTX.with(|thread_ctx| core::ptr::write(thread_ctx.get(), core::ptr::null_mut())); }; log::trace!("Returned from guest program"); - self.set_aux_data_permission_for_host().map_err(Error::from)?; + self.set_aux_data_permission_for_host()?; self.poison = Poison::None; if self.module.as_ref().unwrap().gas_metering() == Some(GasMeteringKind::Async) && self.gas() < 0 { @@ -1740,12 +1971,12 @@ impl super::Sandbox for Sandbox { log::error!("Sandbox poisoned"); InterruptKind::Trap } - ExitReason::Signal => self.handle_guest_signal().map_err(Error::from)?, + ExitReason::Signal => self.handle_guest_signal()?, ExitReason::NotEnoughGas => InterruptKind::NotEnoughGas, ExitReason::Trap => InterruptKind::Trap, ExitReason::Step => InterruptKind::Step, ExitReason::Ecalli(num) => InterruptKind::Ecalli(num), - ExitReason::Segfault(address) => self.handle_guest_pagefault(address).map_err(Error::from)?, + ExitReason::Segfault(address) => self.handle_guest_pagefault(address)?, }) } @@ -1993,10 +2224,10 @@ impl super::Sandbox for Sandbox { debug_assert!(memory_protection.is_none()); } - let Some(slice) = self.get_memory_slice_mut(address, length as u32) else { + let Some(slice) = self.get_memory_slice_mut(address, length) else { return Err(MemoryAccessError::OutOfRangeAccess { address, - length: length as u64, + length: u64::from(length), }); }; @@ -2050,7 +2281,7 @@ impl super::Sandbox for Sandbox { if address <= 0x10000 && length >= 0xffff0000 { self.page_set_present.clear(); self.page_set_writable.clear(); - self.memory.mprotect(self.guest_memory_offset, 0x100000000 as usize, 0)?; + self.memory.mprotect(self.guest_memory_offset, 0x100000000_usize, 0)?; } else { let module = self.module.as_ref().unwrap(); let page_start = module.address_to_page(module.round_to_page_size_down(address)); diff --git a/crates/polkavm/src/sandbox/hypervisor.rs b/crates/polkavm/src/sandbox/hypervisor.rs new file mode 100644 index 000000000..09ca68543 --- /dev/null +++ b/crates/polkavm/src/sandbox/hypervisor.rs @@ -0,0 +1,2232 @@ +//! Hardware-virtualization sandbox backend (aarch64-only). +//! +//! Runs guest code in a vCPU with its own stage-1 MMU, giving 4K guest pages on +//! any host page size (Apple Hypervisor.framework on macOS; KVM on Linux later). +//! +//! Status: the micro-VM engine (`Vm`), the guest `VmCtx` layout / `offset_table` / +//! `address_table`, the memory + register plumbing, `prepare_program` and +//! `load_module` are implemented; `run`/`sbrk` are a first cut. NONE of the guest +//! *execution* path is runtime-verified — that needs a binary signed with the +//! `com.apple.security.hypervisor` entitlement. See the report / NOTEs below. + +use alloc::boxed::Box; +use alloc::sync::Arc; +use alloc::vec; +use alloc::vec::Vec; +use core::mem::MaybeUninit; +use core::sync::atomic::{AtomicI64, AtomicU32, AtomicU64}; + +use polkavm_common::abi::VM_ADDR_RETURN_TO_HOST; +#[cfg(target_os = "macos")] +use polkavm_common::regmap::to_native_reg; +use polkavm_common::zygote::{AddressTable, CacheAligned, VM_ADDR_JUMP_TABLE, VM_ADDR_JUMP_TABLE_RETURN_TO_HOST}; + +use super::{OffsetTable, SandboxInit, SandboxKind}; +use crate::api::{MemoryAccessError, MemoryProtection, Module}; +use crate::compiler::CompiledModule; +use crate::config::{Config, GasMeteringKind}; +use crate::{Gas, InterruptKind, ProgramCounter, Reg, RegValue, Segfault}; + +/// Backend-local error type (mirrors the generic sandbox's `Error`). +#[derive(Debug)] +pub struct Error(alloc::string::String); + +impl core::fmt::Display for Error { + fn fmt(&self, fmt: &mut core::fmt::Formatter) -> core::fmt::Result { + fmt.write_str(&self.0) + } +} + +impl From<&'static str> for Error { + fn from(value: &'static str) -> Self { + Self(value.into()) + } +} + +impl From for Error { + fn from(value: alloc::string::String) -> Self { + Self(value) + } +} + +// --------------------------------------------------------------------------- +// Guest address-space layout (identity: guest VA == guest IPA). +// +// [MAPPED_IPA ................................ MAPPED_IPA + MAPPED_LEN) data (RWX) +// +0 .............. +GUEST_MEM_OFFSET VmCtx page (one 4K page) +// +GUEST_MEM_OFFSET = GUEST_MEM_BASE ....... PolkaVM address 0 (x14 base), 4 GiB +// [STUB_IPA ................................. STUB_IPA + CODE_WINDOW) code (RX) +// +0 ............. +STUB_REGION hvc trampoline stubs (one page) +// +STUB_REGION = NATIVE_CODE_ORIGIN ........ compiled code + inline jump table +// [PT_IPA ...] stage-1 page tables (walker-only) +// +// The VmCtx sits exactly 4096 bytes below the guest memory base because the shared +// AArch64 codegen addresses it as `x14 + GUEST_MEMORY_TO_VMCTX_OFFSET (-4096) + off` +// (see compiler/aarch64.rs). This is only correct when the `generic-sandbox` feature +// is also enabled (the codegen applies that offset only then); the hypervisor test +// matrix always enables both. +// --------------------------------------------------------------------------- + +/// Host (stage-2 / hv_vm_map) page size. Apple Silicon is 16K; all hv_vm_map regions +/// must be aligned to it, even though the guest stage-1 MMU uses a 4K granule. +const HOST_PAGE: usize = 0x4000; +/// Start of the mapped data region in guest IPA space (16K-aligned). +const MAPPED_IPA: u64 = 0x4000_0000; +/// Bytes from the mapped region base to PolkaVM address 0. One host page, so the base +/// stays 16K-aligned; the VmCtx lives in the 4K page just below the base. +const GUEST_MEM_OFFSET: usize = HOST_PAGE; +/// Offset of the VmCtx within the mapped region (exactly 4K below the guest memory base). +const VMCTX_OFFSET: usize = GUEST_MEM_OFFSET - 0x1000; +/// Guest memory base (PolkaVM address 0); held in x14 while the guest runs. +const GUEST_MEM_BASE: u64 = MAPPED_IPA + GUEST_MEM_OFFSET as u64; +/// The 4 GiB PolkaVM address window. +const RAM_SIZE: usize = 0x1_0000_0000; +/// Total host-backed data mapping: VmCtx page + 4 GiB window. +const MAPPED_LEN: usize = GUEST_MEM_OFFSET + RAM_SIZE; +/// Guest page size enforced by the stage-1 MMU (4K granule, TG0 = 0b00). +const GUEST_PAGE_SIZE: usize = 4096; +/// Number of 4K guest pages backing the RAM window. +const NUM_PAGES: usize = RAM_SIZE / GUEST_PAGE_SIZE; + +/// Base of the guest code region. First page: EL1 exception vector (VBAR_EL1) then the +/// hvc trampoline stubs. STUB_IPA is 2048-aligned as VBAR requires. +const STUB_IPA: u64 = 0x1_8000_0000; +/// Guest EL1 exception vector table (16 x 0x80). VBAR_EL1 points here. +const VBAR_IPA: u64 = STUB_IPA; +/// hvc trampoline stubs, placed just after the 2 KiB vector table. +const STUBS_IPA: u64 = STUB_IPA + 0x800; +/// Bytes reserved ahead of the code (vector + stubs), one page. +const STUB_REGION: usize = 0x1000; +/// Where the compiler is told the guest's native code lives. +const NATIVE_CODE_ORIGIN: u64 = STUB_IPA + STUB_REGION as u64; +/// Maximum code+jump-table window we identity-map for a module (512 MiB). +const CODE_WINDOW: usize = 0x2000_0000; +/// End (exclusive) of the IPA range covered by the identity page tables. +const COVER_END: u64 = STUB_IPA + CODE_WINDOW as u64; +/// Number of L1 (1 GiB) entries the identity tables span. +const NUM_L1: usize = (COVER_END - MAPPED_IPA).div_ceil(1 << 30) as usize; + +/// hvc immediates distinguishing the trampoline that trapped (decoded from ESR ISS). +const HVC_HOSTCALL: u16 = 1; +const HVC_TRAP: u16 = 2; +const HVC_RETURN: u16 = 3; +const HVC_STEP: u16 = 4; +const HVC_SBRK: u16 = 5; +const HVC_NOT_ENOUGH_GAS: u16 = 6; +/// Emitted by the guest EL1 exception vector on an abort, to report it to the host. +const HVC_FAULT: u16 = 7; +/// Emitted by the TLB-maintenance stub once the invalidation has completed. +const HVC_TLBI: u16 = 8; +/// Stride between consecutive stubs (2 x 4-byte instructions). +const STUB_STRIDE: u64 = 8; +/// Number of `br`/`blr`-target stubs laid out at `STUBS_IPA` with `STUB_STRIDE` spacing. +const NUM_BR_STUBS: u64 = 6; +/// The TLB-maintenance stub, placed after them (4 instructions, so it needs its own slot). +const TLBI_STUB_IPA: u64 = STUBS_IPA + NUM_BR_STUBS * STUB_STRIDE; + +/// Per-page protection state tracked host-side (mirrors the L3 descriptor AP bits). +const PROT_NONE: u8 = 0; +const PROT_READ: u8 = 1; +const PROT_READ_WRITE: u8 = 2; + +// --------------------------------------------------------------------------- +// Guest VmCtx. +// +// NOTE: this MUST stay bit-identical (field order / types / sizes) to +// `generic::VmCtx`, because the shared AArch64 codegen reads these fields at the +// offsets reported by `offset_table()`, and `emit_gas_metering_stub` hard-asserts +// `gas` lands at 0x60. `maps`/`sandbox` are host-only stand-ins kept only so the +// following fields land at the right offsets. +// --------------------------------------------------------------------------- + +const REG_COUNT: usize = 13; + +#[repr(C)] +struct HeapInfo { + heap_top: u64, + heap_threshold: u64, +} + +#[repr(C)] +#[allow(dead_code)] // variants selected by value when decoding exits +enum HvExitReason { + None, + Error, + Signal, + NotEnoughGas, + Trap, + Ecalli(u32), + Segfault(u64), + Step, +} + +#[repr(C)] +struct VmCtx { + return_address: usize, + return_stack_pointer: usize, + arg: AtomicU32, + heap_info: HeapInfo, + heap_base: u32, + heap_initial_threshold: u32, + heap_max_size: u32, + heap_map_index: usize, + page_size: u32, + maps: Vec, // layout-compatible stand-in for generic's Vec + gas: AtomicI64, + program_range: core::ops::Range, + exit_reason: HvExitReason, + regs: CacheAligned<[RegValue; REG_COUNT]>, + tmp_reg: AtomicU64, + sandbox: *mut (), + program_counter: AtomicU32, + next_program_counter: AtomicU32, + next_native_program_counter: AtomicU64, + memset_continuation: AtomicU64, +} + +impl VmCtx { + fn new() -> Self { + VmCtx { + return_address: 0, + return_stack_pointer: 0, + arg: AtomicU32::new(0), + heap_info: HeapInfo { + heap_top: 0, + heap_threshold: 0, + }, + heap_base: 0, + heap_initial_threshold: 0, + heap_max_size: 0, + heap_map_index: 0, + page_size: 0, + maps: Vec::new(), + gas: AtomicI64::new(0), + program_range: 0..0, + exit_reason: HvExitReason::None, + regs: CacheAligned([0; REG_COUNT]), + tmp_reg: AtomicU64::new(0), + sandbox: core::ptr::null_mut(), + program_counter: AtomicU32::new(0), + next_program_counter: AtomicU32::new(0), + next_native_program_counter: AtomicU64::new(0), + memset_continuation: AtomicU64::new(0), + } + } +} + +polkavm_common::static_assert!(core::mem::size_of::() <= 0x1000); + +// --------------------------------------------------------------------------- +// Hypervisor.framework FFI. +// +// Transcribed from the macOS SDK headers under: +// .../MacOSX.sdk/System/Library/Frameworks/Hypervisor.framework/Versions/A/Headers/ +// (hv_vm.h, hv_vm_types.h, hv_vcpu.h, hv_vcpu_types.h, hv_error.h) +// `hv_return_t` is `mach_error_t` (a C `int`); `HV_SUCCESS == 0`. The ARM memory +// flags live in the (kernel-only) hv_kern_types.h; READ/WRITE/EXEC are 1/2/4. +// --------------------------------------------------------------------------- +#[cfg(target_os = "macos")] +#[allow(non_camel_case_types, dead_code)] // full FFI surface kept for completeness +mod hv { + use core::ffi::c_void; + + pub type hv_return_t = i32; // mach_error_t + pub type hv_vcpu_t = u64; + pub type hv_reg_t = u32; + pub type hv_sys_reg_t = u16; + pub type hv_ipa_t = u64; + pub type hv_memory_flags_t = u64; + pub type hv_exit_reason_t = u32; + + pub const HV_SUCCESS: hv_return_t = 0; + + // hv_memory_flags_t (hv_vm_map / hv_vm_protect). + pub const HV_MEMORY_READ: hv_memory_flags_t = 1 << 0; + pub const HV_MEMORY_WRITE: hv_memory_flags_t = 1 << 1; + pub const HV_MEMORY_EXEC: hv_memory_flags_t = 1 << 2; + + // hv_reg_t (subset we touch). + pub const HV_REG_X0: hv_reg_t = 0; + pub const HV_REG_PC: hv_reg_t = 31; + pub const HV_REG_CPSR: hv_reg_t = 34; + + // hv_sys_reg_t (from hv_vcpu_types.h). + pub const HV_SYS_REG_SCTLR_EL1: hv_sys_reg_t = 0xc080; + pub const HV_SYS_REG_CPACR_EL1: hv_sys_reg_t = 0xc082; + pub const HV_SYS_REG_TTBR0_EL1: hv_sys_reg_t = 0xc100; + pub const HV_SYS_REG_TCR_EL1: hv_sys_reg_t = 0xc102; + pub const HV_SYS_REG_MAIR_EL1: hv_sys_reg_t = 0xc510; + pub const HV_SYS_REG_VBAR_EL1: hv_sys_reg_t = 0xc600; + pub const HV_SYS_REG_SP_EL1: hv_sys_reg_t = 0xe208; + pub const HV_SYS_REG_ESR_EL1: hv_sys_reg_t = 0xc290; + pub const HV_SYS_REG_FAR_EL1: hv_sys_reg_t = 0xc300; + pub const HV_SYS_REG_ELR_EL1: hv_sys_reg_t = 0xc201; + + // hv_exit_reason_t (from hv_vcpu_types.h). + pub const HV_EXIT_REASON_CANCELED: hv_exit_reason_t = 0; + pub const HV_EXIT_REASON_EXCEPTION: hv_exit_reason_t = 1; + pub const HV_EXIT_REASON_VTIMER_ACTIVATED: hv_exit_reason_t = 2; + pub const HV_EXIT_REASON_UNKNOWN: hv_exit_reason_t = 3; + + #[repr(C)] + #[derive(Clone, Copy)] + pub struct hv_vcpu_exit_exception_t { + pub syndrome: u64, + pub virtual_address: u64, + pub physical_address: hv_ipa_t, + } + + #[repr(C)] + #[derive(Clone, Copy)] + pub struct hv_vcpu_exit_t { + pub reason: hv_exit_reason_t, + pub exception: hv_vcpu_exit_exception_t, + } + + #[link(name = "Hypervisor", kind = "framework")] + extern "C" { + pub fn hv_vm_create(config: *mut c_void) -> hv_return_t; + pub fn hv_vm_destroy() -> hv_return_t; + pub fn hv_vm_map(addr: *mut c_void, ipa: hv_ipa_t, size: usize, flags: hv_memory_flags_t) -> hv_return_t; + pub fn hv_vm_unmap(ipa: hv_ipa_t, size: usize) -> hv_return_t; + pub fn hv_vm_protect(ipa: hv_ipa_t, size: usize, flags: hv_memory_flags_t) -> hv_return_t; + + pub fn hv_vcpu_create(vcpu: *mut hv_vcpu_t, exit: *mut *const hv_vcpu_exit_t, config: *mut c_void) -> hv_return_t; + pub fn hv_vcpu_destroy(vcpu: hv_vcpu_t) -> hv_return_t; + pub fn hv_vcpu_run(vcpu: hv_vcpu_t) -> hv_return_t; + pub fn hv_vcpus_exit(vcpus: *const hv_vcpu_t, vcpu_count: u32) -> hv_return_t; + + pub fn hv_vcpu_get_reg(vcpu: hv_vcpu_t, reg: hv_reg_t, value: *mut u64) -> hv_return_t; + pub fn hv_vcpu_set_reg(vcpu: hv_vcpu_t, reg: hv_reg_t, value: u64) -> hv_return_t; + pub fn hv_vcpu_get_sys_reg(vcpu: hv_vcpu_t, reg: hv_sys_reg_t, value: *mut u64) -> hv_return_t; + pub fn hv_vcpu_set_sys_reg(vcpu: hv_vcpu_t, reg: hv_sys_reg_t, value: u64) -> hv_return_t; + } +} + +// --------------------------------------------------------------------------- +// The micro-VM engine. +// --------------------------------------------------------------------------- + +/// Reason a `Vm::run` returned control to the host (decoded from ESR EC). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[allow(dead_code)] +pub enum VmExit { + /// Guest issued `HVC` (EC 0x16): a host-call / return-to-host trampoline; `imm` is the stub id. + HostCall { imm: u16 }, + /// Data/instruction abort (EC 0x24/0x20): a guest memory fault. + MemoryFault { address: u64, esr: u64 }, + /// Undefined instruction (EC 0x00): a guest trap. + Trap { esr: u64 }, + /// Some other synchronous exception. + Exception { ec: u64, esr: u64 }, + /// `hv_vcpus_exit` was called from another thread. + Canceled, + /// The virtual timer fired. + VTimer, + /// The framework could not classify the exit. + Unknown, +} + +// Stage-1 descriptor bits (4K granule). +#[cfg(target_os = "macos")] +const DESC_VALID: u64 = 1 << 0; +#[cfg(target_os = "macos")] +const DESC_PAGE: u64 = 0b11; // valid + page/table +#[cfg(target_os = "macos")] +const PTE_ATTRINDX0: u64 = 0 << 2; +#[cfg(target_os = "macos")] +const PTE_AP_RW_EL1: u64 = 0 << 6; // EL1 read/write, no EL0 +#[cfg(target_os = "macos")] +const PTE_AP_RO_EL1: u64 = 2 << 6; // EL1 read-only, no EL0 +#[cfg(target_os = "macos")] +const PTE_AP_MASK: u64 = 0b11 << 6; +#[cfg(target_os = "macos")] +const PTE_SH_INNER: u64 = 3 << 8; +#[cfg(target_os = "macos")] +const PTE_AF: u64 = 1 << 10; +#[cfg(target_os = "macos")] +const PTE_OUTPUT_MASK: u64 = 0x0000_ffff_ffff_f000; // output address bits [47:12] + +/// A live micro-VM: backing RAM + identity 4K page tables + a code region + one vCPU. +pub struct Vm { + #[cfg(target_os = "macos")] + ram: *mut u8, + #[cfg(target_os = "macos")] + ram_len: usize, + #[cfg(target_os = "macos")] + pt: *mut u8, + #[cfg(target_os = "macos")] + pt_len: usize, + /// IPA at which the page-table region is mapped. + #[cfg(target_os = "macos")] + pt_ipa: u64, + /// Next free table page in the PT region (for on-demand L2/L3 chains). + #[cfg(target_os = "macos")] + pt_next: usize, + /// Total table pages available in the PT region. + #[cfg(target_os = "macos")] + pt_cap: usize, + /// Guest code region (stubs + compiled code), mapped once a module is loaded. + #[cfg(target_os = "macos")] + code: *mut u8, + #[cfg(target_os = "macos")] + code_len: usize, + /// Host backing for the sparse return-to-host jump-table slot page. + #[cfg(target_os = "macos")] + slot: *mut u8, + #[cfg(target_os = "macos")] + slot_len: usize, + #[cfg(target_os = "macos")] + slot_ipa: u64, + #[cfg(target_os = "macos")] + vcpu: hv::hv_vcpu_t, + /// Whether `vcpu` holds a live vCPU (its id can legitimately be 0, so no sentinel). + #[cfg(target_os = "macos")] + has_vcpu: bool, + #[cfg(target_os = "macos")] + exit: *const hv::hv_vcpu_exit_t, + /// Set when the host edits the stage-1 tables; cleared by `flush_tlb` before the next entry. + #[cfg(target_os = "macos")] + tlb_dirty: bool, + /// Held for the VM's whole lifetime: only one HV VM may exist per process, so + /// concurrent sandboxes serialize here instead of failing. Released on drop. + #[cfg(target_os = "macos")] + _guard: std::sync::MutexGuard<'static, ()>, +} + +// SAFETY: the raw pointers refer to this VM's own mappings; ownership moves with the box. +unsafe impl Send for Vm {} +// SAFETY: guest RAM/page tables are plain memory; a vCPU is only ever driven from the +// owning instance's thread. Mirrors the generic sandbox's Send+Sync treatment of `Mmap`. +unsafe impl Sync for Vm {} + +/// How long a `spawn` waits for a live instance to be dropped before giving up. +#[cfg(target_os = "macos")] +const VM_ACQUIRE_TIMEOUT: core::time::Duration = core::time::Duration::from_secs(5); + +/// Serializes VM creation/teardown: Hypervisor.framework allows only one VM per process. +#[cfg(target_os = "macos")] +static VM_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + +#[cfg(target_os = "macos")] +thread_local! { + /// Whether the current thread owns the live VM (its vCPU is bound to this thread). + static VM_HELD: core::cell::Cell = const { core::cell::Cell::new(false) }; +} + +impl Vm { + /// Create the VM: mmap RAM, create the HV VM, map RAM + page tables, build the + /// identity 4K tables, create the vCPU and inject the MMU sysregs. + pub fn new() -> Result { + #[cfg(target_os = "macos")] + { + // One VM per process, one vCPU per thread: fail cleanly instead of self-deadlocking. + let start = std::time::Instant::now(); + if VM_HELD.with(core::cell::Cell::get) { + return Err("hypervisor sandbox: this thread already owns the single per-process VM; \ + drop the previous instance before creating another (nested/simultaneous instances are unsupported)" + .into()); + } + // Bounded wait: the owner may be the caller's own live instance, which never yields. + let guard = loop { + match VM_LOCK.try_lock() { + Ok(guard) => break guard, + Err(std::sync::TryLockError::Poisoned(error)) => break error.into_inner(), + Err(std::sync::TryLockError::WouldBlock) => { + if start.elapsed() >= VM_ACQUIRE_TIMEOUT { + return Err("hypervisor sandbox: timed out waiting for the single per-process VM \ + (Hypervisor.framework allows one VM per process and one vCPU per thread, so \ + instances cannot overlap)" + .into()); + } + std::thread::sleep(core::time::Duration::from_millis(1)); + } + } + }; + VM_HELD.with(|h| h.set(true)); + let result = Self::new_macos(guard); + if result.is_err() { + VM_HELD.with(|h| h.set(false)); + } + result + } + #[cfg(not(target_os = "macos"))] + { + // TODO(hypervisor): implement the KVM-backed engine for aarch64-linux. + todo!("hypervisor sandbox: no Hypervisor.framework on this OS (KVM backend not yet implemented)") + } + } + + #[cfg(target_os = "macos")] + fn new_macos(guard: std::sync::MutexGuard<'static, ()>) -> Result { + // Cover [MAPPED_IPA, COVER_END): one L1 table, one L2 per 1 GiB, one L3 per 2 MiB. + let l1_span = 1u64 << 30; + let num_l1 = (COVER_END - MAPPED_IPA).div_ceil(l1_span) as usize; + let num_l3 = num_l1 * 512; + let identity_tables = 1 + num_l1 + num_l3; + // A few spare table pages back on-demand chains (e.g. the sparse return slot). + const SPARE_TABLES: usize = 8; + let pt_len = align_up((identity_tables + SPARE_TABLES) * GUEST_PAGE_SIZE, HOST_PAGE); + let pt_cap = pt_len / GUEST_PAGE_SIZE; + let pt_ipa = COVER_END; // walker-only; needs no stage-1 mapping + + let ram = map_anon(MAPPED_LEN)?; + let pt = match map_anon(pt_len) { + Ok(pt) => pt, + Err(error) => { + unmap_anon(ram, MAPPED_LEN); + return Err(error); + } + }; + + let mut vm = Vm { + ram, + ram_len: MAPPED_LEN, + pt, + pt_len, + pt_ipa, + pt_next: identity_tables, + pt_cap, + code: core::ptr::null_mut(), + code_len: 0, + slot: core::ptr::null_mut(), + slot_len: 0, + slot_ipa: 0, + vcpu: 0, + has_vcpu: false, + exit: core::ptr::null(), + tlb_dirty: false, + _guard: guard, + }; + + if let Err(error) = vm.bringup(num_l1) { + vm.teardown(); + return Err(error); + } + + Ok(vm) + } + + #[cfg(target_os = "macos")] + fn bringup(&mut self, num_l1: usize) -> Result<(), Error> { + // Create the per-process VM. + // SAFETY: FFI call; a null config selects the framework default. + check(unsafe { hv::hv_vm_create(core::ptr::null_mut()) }, "hv_vm_create")?; + + // Map the data region (RW only; Apple's Hypervisor.framework forbids W+X). Code + // is mapped R+X separately in `load_code`. + check( + // SAFETY: `self.ram`/`self.ram_len` describe a live anonymous mapping. + unsafe { hv::hv_vm_map(self.ram.cast(), MAPPED_IPA, self.ram_len, hv::HV_MEMORY_READ | hv::HV_MEMORY_WRITE) }, + "hv_vm_map(ram)", + )?; + // Map the page-table region (RW). + check( + // SAFETY: `self.pt`/`self.pt_len` describe a live anonymous mapping. + unsafe { hv::hv_vm_map(self.pt.cast(), self.pt_ipa, self.pt_len, hv::HV_MEMORY_READ | hv::HV_MEMORY_WRITE) }, + "hv_vm_map(page-tables)", + )?; + + self.build_identity_4k_tables(num_l1); + + // Create the vCPU on the current thread. + let mut vcpu: hv::hv_vcpu_t = 0; + let mut exit: *const hv::hv_vcpu_exit_t = core::ptr::null(); + check( + // SAFETY: FFI call; `vcpu`/`exit` are valid out-pointers, null config = default. + unsafe { hv::hv_vcpu_create(&mut vcpu, &mut exit, core::ptr::null_mut()) }, + "hv_vcpu_create", + )?; + self.vcpu = vcpu; + self.has_vcpu = true; + self.exit = exit; + + self.inject_mmu_sysregs()?; + Ok(()) + } + + /// Build identity 4K page tables covering `[MAPPED_IPA, MAPPED_IPA + num_l1 GiB)`. + /// + /// Layout in the PT region: `[L1][L2 x num_l1][L3 x (num_l1 * 512)]`, contiguous. + // The PT region is `mmap`-allocated and thus page-aligned, so the `*mut u8` -> `*mut u64` + // casts and every 8-byte-aligned table offset within it are correctly aligned. + #[cfg(target_os = "macos")] + #[allow(clippy::cast_ptr_alignment)] + fn build_identity_4k_tables(&mut self, num_l1: usize) { + let l1_start_idx = (MAPPED_IPA >> 30) as usize & 0x1ff; + let table_ipa = |table_index: usize| self.pt_ipa + (table_index as u64) * GUEST_PAGE_SIZE as u64; + + // L1 table is table 0; L2 tables are 1..=num_l1; L3 tables follow. + let l1 = self.pt.cast::(); + for i in 0..num_l1 { + let entry = table_ipa(1 + i) | DESC_PAGE; + // SAFETY: `l1_start_idx + i` is a valid entry index within the L1 table page. + unsafe { l1.add(l1_start_idx + i).write(entry) }; + } + + for i in 0..num_l1 { + // SAFETY: L2 table `i` lives at table offset `1 + i`, inside the PT region. + let l2 = unsafe { self.pt.add((1 + i) * GUEST_PAGE_SIZE).cast::() }; + for j in 0..512usize { + let entry = table_ipa(1 + num_l1 + i * 512 + j) | DESC_PAGE; + // SAFETY: `j` is a valid entry index (0..512) within this L2 table page. + unsafe { l2.add(j).write(entry) }; + } + } + + for t in 0..(num_l1 * 512) { + // SAFETY: L3 table `t` lives at table offset `1 + num_l1 + t`, inside the PT region. + let l3 = unsafe { self.pt.add((1 + num_l1 + t) * GUEST_PAGE_SIZE).cast::() }; + for m in 0..512usize { + let page_va = MAPPED_IPA + ((t * 512 + m) * GUEST_PAGE_SIZE) as u64; + let entry = (page_va & PTE_OUTPUT_MASK) | PTE_AF | PTE_SH_INNER | PTE_AP_RW_EL1 | PTE_ATTRINDX0 | DESC_PAGE; + // SAFETY: `m` is a valid entry index (0..512) within this L3 table page. + unsafe { l3.add(m).write(entry) }; + } + } + } + + /// Invalidate every L3 descriptor covering the PolkaVM data window so unmapped + /// accesses fault (dynamic paging). Iterates the L3 tables directly for speed. + // The PT region is page-aligned so the `*mut u8` -> `*mut u64` casts are aligned. + #[cfg(target_os = "macos")] + #[allow(clippy::cast_ptr_alignment)] + fn clear_window(&mut self) { + self.tlb_dirty = true; + let start = GUEST_MEM_OFFSET / GUEST_PAGE_SIZE; // covered-page index of PolkaVM address 0 + let end = start + NUM_PAGES; + for p in start..end { + // SAFETY: covered page `p` maps to L3 table `1 + NUM_L1 + p/512`, inside the PT region. + let l3 = unsafe { self.pt.add((1 + NUM_L1 + p / 512) * GUEST_PAGE_SIZE).cast::() }; + // SAFETY: `p % 512` is a valid entry index in that table. + unsafe { + let e = l3.add(p % 512).read(); + l3.add(p % 512).write(e & !DESC_VALID); + } + } + } + + /// Grab a spare, zeroed table page from the PT region; returns (host ptr, IPA). + // The PT region is page-aligned so the `*mut u8` -> `*mut u64` cast is aligned. + #[cfg(target_os = "macos")] + #[allow(clippy::cast_ptr_alignment)] + fn alloc_table_page(&mut self) -> Result<(*mut u64, u64), Error> { + if self.pt_next >= self.pt_cap { + return Err("out of spare page-table pages".into()); + } + let idx = self.pt_next; + self.pt_next += 1; + let ipa = self.pt_ipa + (idx * GUEST_PAGE_SIZE) as u64; + // SAFETY: `idx < pt_cap`, so the page lies within the PT mapping; mmap zeroed it. + let host = unsafe { self.pt.add(idx * GUEST_PAGE_SIZE).cast::() }; + Ok((host, ipa)) + } + + /// Map a single read-only guest page holding `value` at `slot_va`, wiring an + /// on-demand L1->L2->L3 chain. Used for the sparse return-to-host jump-table slot. + // The PT region is page-aligned so the `*mut u8` -> `*mut u64` casts are aligned. + #[cfg(target_os = "macos")] + #[allow(clippy::cast_ptr_alignment)] + fn map_return_slot(&mut self, slot_va: u64, value: u64) -> Result<(), Error> { + self.tlb_dirty = true; + // Back the containing host page and map it read-only into the guest IPA space. + let page_base = slot_va & !(HOST_PAGE as u64 - 1); + if self.slot.is_null() { + self.slot = map_anon(HOST_PAGE)?; + self.slot_len = HOST_PAGE; + self.slot_ipa = page_base; + check( + // SAFETY: `self.slot`/HOST_PAGE describe a live anonymous mapping. + unsafe { hv::hv_vm_map(self.slot.cast(), page_base, HOST_PAGE, hv::HV_MEMORY_READ) }, + "hv_vm_map(return-slot)", + )?; + } + // SAFETY: offset is within the freshly-mapped HOST_PAGE region. + unsafe { + self.slot.add((slot_va - page_base) as usize).cast::().write_unaligned(value); + } + + // Wire L1 -> L2 -> L3 for the 4K guest page containing `slot_va`. + let l1 = self.pt.cast::(); + let l1i = (slot_va >> 30) as usize & 0x1ff; + // SAFETY: `l1i < 512`; L1 table is the first PT page. + let l1d = unsafe { l1.add(l1i).read() }; + let l2 = if l1d & DESC_VALID == 0 { + let (host, ipa) = self.alloc_table_page()?; + // SAFETY: `l1i` in range; installing the new L2 table descriptor. + unsafe { l1.add(l1i).write(ipa | DESC_PAGE) }; + host + } else { + // SAFETY: descriptor output is an IPA inside the PT region. + unsafe { self.pt.add(((l1d & PTE_OUTPUT_MASK) - self.pt_ipa) as usize).cast::() } + }; + let l2i = (slot_va >> 21) as usize & 0x1ff; + // SAFETY: `l2i < 512`; `l2` is a valid table page. + let l2d = unsafe { l2.add(l2i).read() }; + let l3 = if l2d & DESC_VALID == 0 { + let (host, ipa) = self.alloc_table_page()?; + // SAFETY: `l2i` in range; installing the new L3 table descriptor. + unsafe { l2.add(l2i).write(ipa | DESC_PAGE) }; + host + } else { + // SAFETY: descriptor output is an IPA inside the PT region. + unsafe { self.pt.add(((l2d & PTE_OUTPUT_MASK) - self.pt_ipa) as usize).cast::() } + }; + let l3i = (slot_va >> 12) as usize & 0x1ff; + let entry = (slot_va & PTE_OUTPUT_MASK) | PTE_AF | PTE_SH_INNER | PTE_AP_RO_EL1 | PTE_ATTRINDX0 | DESC_PAGE; + // SAFETY: `l3i < 512`; `l3` is a valid table page. + unsafe { l3.add(l3i).write(entry) }; + Ok(()) + } + + /// Program the vCPU's stage-1 MMU and drop it into EL1h with the MMU enabled. + #[cfg(target_os = "macos")] + fn inject_mmu_sysregs(&mut self) -> Result<(), Error> { + // TCR_EL1: T0SZ=25 (39-bit VA), IRGN0/ORGN0 = WB-WA, SH0 = inner-shareable, + // TG0 = 0b00 (4K, so no TG0 bits set), EPD1 = 1 (disable TTBR1), IPS = 2 (40-bit PA). + let tcr: u64 = 25 | (1 << 8) | (1 << 10) | (3 << 12) | (1 << 23) | (2 << 32); + // MAIR: attr0 = Normal, inner/outer write-back non-transient (0xFF). + let mair: u64 = 0xFF; + + self.set_sys_reg(hv::HV_SYS_REG_MAIR_EL1, mair)?; + self.set_sys_reg(hv::HV_SYS_REG_TCR_EL1, tcr)?; + self.set_sys_reg(hv::HV_SYS_REG_TTBR0_EL1, self.pt_ipa)?; + self.set_sys_reg(hv::HV_SYS_REG_VBAR_EL1, VBAR_IPA)?; + // Hardware SP scratch for the trampolines (e.g. sbrk's push/pop): the RW region + // just below the VmCtx, growing down. Distinct from the guest's Reg::SP. + self.set_sys_reg(hv::HV_SYS_REG_SP_EL1, MAPPED_IPA + VMCTX_OFFSET as u64)?; + // Allow FP/SIMD at EL0/EL1 (FPEN = 0b11). + self.set_sys_reg(hv::HV_SYS_REG_CPACR_EL1, 3 << 20)?; + + // SCTLR_EL1: keep reset value, add M (MMU), C (data cache), I (instr cache). + let sctlr = self.get_sys_reg(hv::HV_SYS_REG_SCTLR_EL1)?; + self.set_sys_reg(hv::HV_SYS_REG_SCTLR_EL1, sctlr | 1 | (1 << 2) | (1 << 12))?; + + // CPSR: EL1h with DAIF masked. + self.set_reg(hv::HV_REG_CPSR, 0x3c5)?; + Ok(()) + } + + /// Map the code region (stubs + compiled code) at STUB_IPA as RX and install the + /// hvc trampoline stubs. `code` is the compiled machine code (+ inline jump table). + #[cfg(target_os = "macos")] + fn load_code(&mut self, code: &[u8]) -> Result<(), Error> { + let total = STUB_REGION + code.len(); + if total > CODE_WINDOW { + return Err("compiled code exceeds the mapped code window".into()); + } + let map_len = align_up(total, HOST_PAGE); // hv_vm_map needs 16K multiples + + // Reuse the existing mapping if it is large enough, otherwise (re)allocate. + if self.code.is_null() || map_len > self.code_len { + if !self.code.is_null() { + // SAFETY: unmapping our own live code region before replacing it. + unsafe { hv::hv_vm_unmap(STUB_IPA, self.code_len) }; + unmap_anon(self.code, self.code_len); + self.code = core::ptr::null_mut(); + self.code_len = 0; + } + let ptr = map_anon(map_len)?; + check( + // SAFETY: `ptr`/`map_len` describe a live anonymous mapping. + unsafe { hv::hv_vm_map(ptr.cast(), STUB_IPA, map_len, hv::HV_MEMORY_READ | hv::HV_MEMORY_EXEC) }, + "hv_vm_map(code)", + )?; + self.code = ptr; + self.code_len = map_len; + } + + // Install the hvc stubs in the first page and copy the code after it. + self.install_stubs(); + // SAFETY: `code.len() + STUB_REGION <= map_len`; destination is within the mapping. + unsafe { + core::ptr::copy_nonoverlapping(code.as_ptr(), self.code.add(STUB_REGION), code.len()); + // Keep the guest instruction cache coherent with the writes above. + sys_dcache_flush(self.code.cast(), map_len); + sys_icache_invalidate(self.code.cast(), map_len); + } + Ok(()) + } + + /// Write the EL1 exception vector and the `hvc`-based trampoline stubs into the + /// first code page. + #[cfg(target_os = "macos")] + fn install_stubs(&mut self) { + let hvc = |imm: u16| 0xD400_0002u32 | (u32::from(imm) << 5); // hvc #imm + + // Exception vector: every 0x80-aligned entry does `hvc #FAULT` so any guest + // EL1 exception (esp. a stage-1 data/instruction abort) is reported to the host. + let vbar_off = (VBAR_IPA - STUB_IPA) as usize; + for entry in 0..16usize { + let off = vbar_off + entry * 0x80; + // SAFETY: 16 * 0x80 = 2 KiB, within the reserved stub page. + unsafe { self.code.add(off).cast::().write_unaligned(hvc(HVC_FAULT)) }; + } + + // br-targets (hostcall/trap/return/step/not_enough_gas): a single `hvc #imm`; + // the host stops the run loop and re-enters later via sysenter. sbrk is a + // blr-target so it needs `hvc #imm; ret` to return into its trampoline. + let ret = 0xD65F03C0u32; // ret x30 + let stubs = [ + (HVC_HOSTCALL, false), + (HVC_TRAP, false), + (HVC_RETURN, false), + (HVC_STEP, false), + (HVC_SBRK, true), + (HVC_NOT_ENOUGH_GAS, false), + ]; + let stubs_off = (STUBS_IPA - STUB_IPA) as usize; + debug_assert_eq!(stubs.len() as u64, NUM_BR_STUBS); + for (nth, (imm, needs_ret)) in stubs.iter().enumerate() { + let off = stubs_off + nth * STUB_STRIDE as usize; + // SAFETY: the stubs occupy STUBS_IPA.. within the reserved stub page. + unsafe { + self.code.add(off).cast::().write_unaligned(hvc(*imm)); + let second = if *needs_ret { ret } else { 0x0000_0000u32 }; // udf #0 filler + self.code.add(off + 4).cast::().write_unaligned(second); + } + } + + // TLB maintenance; the host enters here after editing the page tables (see `flush_tlb`). + const TLBI_VMALLE1: u32 = 0xD508_871F; + const DSB_ISH: u32 = 0xD503_3B9F; + const ISB: u32 = 0xD503_3FDF; + let tlbi_off = (TLBI_STUB_IPA - STUB_IPA) as usize; + // SAFETY: 4 instructions at TLBI_STUB_IPA, still within the reserved stub page. + unsafe { + for (nth, word) in [TLBI_VMALLE1, DSB_ISH, ISB, hvc(HVC_TLBI)].into_iter().enumerate() { + self.code.add(tlbi_off + nth * 4).cast::().write_unaligned(word); + } + } + } + + /// Locate the L3 descriptor for a guest VA by walking the tables in host memory. + // The PT region is page-aligned, so the `*mut u8` -> `*mut u64` casts are aligned. + #[cfg(target_os = "macos")] + #[allow(clippy::cast_ptr_alignment)] + unsafe fn l3_entry_ptr(&self, va: u64) -> *mut u64 { + let host_of_ipa = |ipa: u64| self.pt.add((ipa - self.pt_ipa) as usize).cast::(); + + let l1 = self.pt.cast::(); + let l1d = l1.add((va >> 30) as usize & 0x1ff).read(); + let l2 = host_of_ipa(l1d & PTE_OUTPUT_MASK); + let l2d = l2.add((va >> 21) as usize & 0x1ff).read(); + let l3 = host_of_ipa(l2d & PTE_OUTPUT_MASK); + l3.add((va >> 12) as usize & 0x1ff) + } + + /// Flip the AP bits of the L3 descriptors for `[va, va + len)` (PolkaVM addresses). + /// This is where the 4K granule matters: protections are per 4K page. + #[cfg(target_os = "macos")] + fn set_page_protection(&mut self, va: u32, len: u32, ap_bits: u64) { + if len == 0 { + return; + } + self.tlb_dirty = true; + let first = u64::from(va) & !(GUEST_PAGE_SIZE as u64 - 1); + let last = (u64::from(va) + u64::from(len) - 1) & !(GUEST_PAGE_SIZE as u64 - 1); + let mut page_va = first; + while page_va <= last { + // SAFETY: `page_va` maps into the identity-mapped window, so the walk resolves + // to a real L3 descriptor in the PT region. + let entry_ptr = unsafe { self.l3_entry_ptr(GUEST_MEM_BASE + page_va) }; + // SAFETY: `entry_ptr` points at a live, aligned L3 descriptor. + let entry = unsafe { entry_ptr.read() }; + let entry = (entry & !PTE_AP_MASK) | (ap_bits & PTE_AP_MASK) | DESC_PAGE; + // SAFETY: same descriptor pointer; writing back the updated entry. + unsafe { entry_ptr.write(entry) }; + page_va += GUEST_PAGE_SIZE as u64; + } + } + + /// Invalidate (make faulting) the L3 descriptors for `[va, va + len)`. + #[cfg(target_os = "macos")] + fn invalidate_pages(&mut self, va: u32, len: u32) { + if len == 0 { + return; + } + self.tlb_dirty = true; + let first = u64::from(va) & !(GUEST_PAGE_SIZE as u64 - 1); + let last = (u64::from(va) + u64::from(len) - 1) & !(GUEST_PAGE_SIZE as u64 - 1); + let mut page_va = first; + while page_va <= last { + // SAFETY: `page_va` maps into the identity-mapped window; the walk resolves + // to a real L3 descriptor in the PT region. + let entry_ptr = unsafe { self.l3_entry_ptr(GUEST_MEM_BASE + page_va) }; + // SAFETY: `entry_ptr` points at a live, aligned L3 descriptor. + let entry = unsafe { entry_ptr.read() }; + // SAFETY: same descriptor pointer; clearing the valid bit. + unsafe { entry_ptr.write(entry & !DESC_VALID) }; + page_va += GUEST_PAGE_SIZE as u64; + } + } + + /// Invalidate the guest's stage-1 TLB if the host edited the page tables. Only the guest can do + /// it, so enter briefly at the TLBI stub: it touches no GPRs and the caller resets PC anyway. + #[cfg(target_os = "macos")] + fn flush_tlb(&mut self) -> Result<(), Error> { + if !self.tlb_dirty { + return Ok(()); + } + self.set_reg(hv::HV_REG_PC, TLBI_STUB_IPA)?; + match self.run_raw()? { + VmExit::HostCall { imm: HVC_TLBI } => { + self.tlb_dirty = false; + Ok(()) + } + other => Err(alloc::format!("unexpected exit while invalidating the guest TLB: {other:?}").into()), + } + } + + /// Host pointer to the guest VmCtx (one 4K page below the guest memory base). + #[cfg(target_os = "macos")] + #[allow(clippy::cast_ptr_alignment)] // `ram` is page-aligned from mmap; VMCTX_OFFSET is 4K-aligned + fn vmctx_ptr(&self) -> *mut VmCtx { + // SAFETY: `VMCTX_OFFSET` is within the mapped region. + unsafe { self.ram.add(VMCTX_OFFSET).cast::() } + } + + /// Run the vCPU and decode the exit reason. + #[allow(dead_code)] + pub fn run_raw(&mut self) -> Result { + #[cfg(target_os = "macos")] + { + // SAFETY: FFI call on this thread's own vCPU. + check(unsafe { hv::hv_vcpu_run(self.vcpu) }, "hv_vcpu_run")?; + // SAFETY: `exit` points at framework-owned storage updated by hv_vcpu_run. + let exit = unsafe { &*self.exit }; + let vm_exit = match exit.reason { + hv::HV_EXIT_REASON_CANCELED => VmExit::Canceled, + hv::HV_EXIT_REASON_VTIMER_ACTIVATED => VmExit::VTimer, + hv::HV_EXIT_REASON_UNKNOWN => VmExit::Unknown, + hv::HV_EXIT_REASON_EXCEPTION => { + let esr = exit.exception.syndrome; + let ec = (esr >> 26) & 0x3f; + match ec { + 0x16 => VmExit::HostCall { + imm: (esr & 0xffff) as u16, + }, // HVC + 0x24 | 0x20 => VmExit::MemoryFault { + address: exit.exception.virtual_address, + esr, + }, // Data / Instruction Abort + 0x00 => VmExit::Trap { esr }, // Undefined instruction + _ => VmExit::Exception { ec, esr }, + } + } + _ => VmExit::Unknown, + }; + Ok(vm_exit) + } + #[cfg(not(target_os = "macos"))] + { + todo!("hypervisor sandbox: KVM run loop not yet implemented") + } + } + + #[allow(dead_code)] + pub fn get_reg(&self, reg: u32) -> Result { + #[cfg(target_os = "macos")] + { + let mut value = 0u64; + // SAFETY: FFI call; `value` is a valid out-pointer. + check(unsafe { hv::hv_vcpu_get_reg(self.vcpu, reg, &mut value) }, "hv_vcpu_get_reg")?; + Ok(value) + } + #[cfg(not(target_os = "macos"))] + { + let _ = reg; + todo!("hypervisor sandbox: KVM get_reg not yet implemented") + } + } + + #[allow(dead_code)] + pub fn set_reg(&mut self, reg: u32, value: u64) -> Result<(), Error> { + #[cfg(target_os = "macos")] + { + // SAFETY: FFI call on this thread's own vCPU. + check(unsafe { hv::hv_vcpu_set_reg(self.vcpu, reg, value) }, "hv_vcpu_set_reg") + } + #[cfg(not(target_os = "macos"))] + { + let _ = (reg, value); + todo!("hypervisor sandbox: KVM set_reg not yet implemented") + } + } + + #[allow(dead_code)] + pub fn get_sys_reg(&self, reg: u16) -> Result { + #[cfg(target_os = "macos")] + { + let mut value = 0u64; + // SAFETY: FFI call; `value` is a valid out-pointer. + let ret = unsafe { hv::hv_vcpu_get_sys_reg(self.vcpu, reg, &mut value) }; + check(ret, "hv_vcpu_get_sys_reg")?; + Ok(value) + } + #[cfg(not(target_os = "macos"))] + { + let _ = reg; + todo!("hypervisor sandbox: KVM get_sys_reg not yet implemented") + } + } + + #[allow(dead_code)] + pub fn set_sys_reg(&mut self, reg: u16, value: u64) -> Result<(), Error> { + #[cfg(target_os = "macos")] + { + // SAFETY: FFI call on this thread's own vCPU. + check(unsafe { hv::hv_vcpu_set_sys_reg(self.vcpu, reg, value) }, "hv_vcpu_set_sys_reg") + } + #[cfg(not(target_os = "macos"))] + { + let _ = (reg, value); + todo!("hypervisor sandbox: KVM set_sys_reg not yet implemented") + } + } + + /// Host-side view of guest RAM at PolkaVM `[addr, addr + len)`. + #[cfg(target_os = "macos")] + fn ram_slice(&self, addr: u32, len: usize) -> Option<&[u8]> { + let end = u64::from(addr).checked_add(len as u64)?; + if end > RAM_SIZE as u64 { + return None; + } + // SAFETY: bounds checked against the 4 GiB window; skip the VmCtx page. + Some(unsafe { core::slice::from_raw_parts(self.ram.add(GUEST_MEM_OFFSET + addr as usize), len) }) + } + + #[cfg(target_os = "macos")] + fn ram_slice_mut(&mut self, addr: u32, len: usize) -> Option<&mut [u8]> { + let end = u64::from(addr).checked_add(len as u64)?; + if end > RAM_SIZE as u64 { + return None; + } + // SAFETY: bounds checked against the 4 GiB window; skip the VmCtx page. + Some(unsafe { core::slice::from_raw_parts_mut(self.ram.add(GUEST_MEM_OFFSET + addr as usize), len) }) + } + + #[cfg(target_os = "macos")] + fn teardown(&mut self) { + // vCPU (if created) must be destroyed before the VM. + if self.has_vcpu { + // SAFETY: FFI call; destroying this thread's own vCPU. + unsafe { hv::hv_vcpu_destroy(self.vcpu) }; + self.has_vcpu = false; + } + // SAFETY: FFI call; all vCPUs have been destroyed above. + unsafe { hv::hv_vm_destroy() }; + if !self.ram.is_null() { + unmap_anon(self.ram, self.ram_len); + self.ram = core::ptr::null_mut(); + } + if !self.pt.is_null() { + unmap_anon(self.pt, self.pt_len); + self.pt = core::ptr::null_mut(); + } + if !self.code.is_null() { + unmap_anon(self.code, self.code_len); + self.code = core::ptr::null_mut(); + } + if !self.slot.is_null() { + unmap_anon(self.slot, self.slot_len); + self.slot = core::ptr::null_mut(); + } + // Release this thread's ownership; the VM_LOCK guard drops after this returns. + VM_HELD.with(|h| h.set(false)); + } +} + +impl Drop for Vm { + fn drop(&mut self) { + #[cfg(target_os = "macos")] + { + if !self.ram.is_null() { + self.teardown(); + } + } + } +} + +/// mmap an anonymous, private, read-write host region. +#[cfg(target_os = "macos")] +fn map_anon(len: usize) -> Result<*mut u8, Error> { + // SAFETY: standard anonymous mmap; result is checked below. + let ptr = unsafe { + libc::mmap( + core::ptr::null_mut(), + len, + libc::PROT_READ | libc::PROT_WRITE, + libc::MAP_ANON | libc::MAP_PRIVATE, + -1, + 0, + ) + }; + if ptr == libc::MAP_FAILED { + return Err("mmap failed".into()); + } + Ok(ptr.cast()) +} + +#[cfg(target_os = "macos")] +fn unmap_anon(ptr: *mut u8, len: usize) { + // SAFETY: `ptr`/`len` come from a matching `map_anon`. + unsafe { libc::munmap(ptr.cast(), len) }; +} + +#[cfg(target_os = "macos")] +fn check(ret: hv::hv_return_t, what: &str) -> Result<(), Error> { + if ret == hv::HV_SUCCESS { + Ok(()) + } else { + Err(alloc::format!("{what} failed: hv_return_t=0x{:08x}", ret as u32).into()) + } +} + +#[cfg(target_os = "macos")] +fn align_up(value: usize, align: usize) -> usize { + (value + align - 1) & !(align - 1) +} + +// Apple's cache-maintenance helpers (from ). Signatures match +// the generic sandbox's declaration to avoid a cross-module redeclaration clash. +#[cfg(target_os = "macos")] +extern "C" { + fn sys_icache_invalidate(start: *mut core::ffi::c_void, len: libc::size_t); + fn sys_dcache_flush(start: *mut core::ffi::c_void, len: libc::size_t); +} + +// --------------------------------------------------------------------------- +// Sandbox trait plumbing. +// --------------------------------------------------------------------------- + +/// Per-engine state shared across sandboxes. +pub struct GlobalState {} + +impl GlobalState { + pub fn new(_config: &Config) -> Result { + Ok(GlobalState {}) + } +} + +#[derive(Default)] +pub struct SandboxConfig {} + +impl super::SandboxConfig for SandboxConfig { + fn enable_logger(&mut self, _value: bool) {} + fn enable_sandboxing(&mut self, _value: bool) {} +} + +/// One region to materialize into guest RAM when a module is loaded. +#[derive(Clone)] +struct ProgramMap { + address: u32, + length: u32, + is_writable: bool, + /// Initial bytes; `None` means zero-filled. + data: Option>, +} + +/// A prepared program: guest machine code + jump table + memory layout. +#[derive(Clone)] +pub struct SandboxProgram(Arc); + +struct SandboxProgramInner { + /// Compiled code followed by the inline jump table (as emitted by the compiler). + code: Vec, + /// Raw jump-table bytes (already appended to `code` by the compiler pipeline layout). + jump_table: Vec, + sysenter_address: u64, + // Written into the return-to-host jump-table slot (see load_module). + sysreturn_address: u64, + memory_map: Vec, + heap_map_index: usize, +} + +impl super::SandboxProgram for SandboxProgram { + fn machine_code(&self) -> &[u8] { + &self.0.code + } +} + +/// Reserved guest address space backing the stage-1 MMU. +pub struct AddressSpace { + native_code_origin: u64, +} + +impl super::SandboxAddressSpace for AddressSpace { + fn native_code_origin(&self) -> u64 { + self.native_code_origin + } +} + +/// A live sandbox: a vCPU with its own guest memory. +pub struct Sandbox { + vm: Vm, + module: Option, + program: Option, + gas_metering: Option, + regs: [RegValue; REG_COUNT], + gas: Gas, + program_counter: Option, + is_program_counter_valid: bool, + next_program_counter: Option, + next_program_counter_changed: bool, + charge_gas_on_entry: bool, + /// Page size faults are reported at; descriptors stay 4K, so any `GUEST_PAGE_SIZE` multiple works. + page_size: u32, + /// Aux data the guest may read, as set by `set_accessible_aux_size`. + accessible_aux_size: u32, + aux_data_address: u32, + aux_data_full_length: u32, + heap_base: u32, + heap_top: u32, + dynamic_paging: bool, + /// Per-4K-page protection state, indexed by page number within the RAM window. + page_prot: Vec, +} + +impl Drop for Sandbox { + fn drop(&mut self) {} +} + +/// Bounds-check a `[address, address + length)` access against the 4 GiB RAM window. +fn bounds_ok(address: u32, length: u32) -> bool { + u64::from(address) + u64::from(length) <= RAM_SIZE as u64 +} + +impl super::Sandbox for Sandbox { + const KIND: SandboxKind = SandboxKind::Hypervisor; + + type Config = SandboxConfig; + type Error = Error; + type Program = SandboxProgram; + type AddressSpace = AddressSpace; + type GlobalState = GlobalState; + type JumpTable = Vec; + + fn downcast_module(module: &Module) -> &CompiledModule { + #[allow(unreachable_patterns, clippy::match_wildcard_for_single_variants)] + match module.compiled_module() { + crate::api::CompiledModuleKind::Hypervisor(ref module) => module, + _ => unreachable!(), + } + } + + fn downcast_global_state(global: &crate::sandbox::GlobalStateKind) -> &Self::GlobalState { + #[allow(unreachable_patterns, clippy::match_wildcard_for_single_variants)] + match global { + crate::sandbox::GlobalStateKind::Hypervisor(global) => global, + _ => unreachable!(), + } + } + + fn allocate_jump_table(_global: &Self::GlobalState, count: usize) -> Result { + // Byte size must be a whole number of native pages (see compiler.rs finish_compilation). + let page = crate::sandbox::get_native_page_size(); + let size = polkavm_common::utils::align_to_next_page_usize(page, count * core::mem::size_of::()).unwrap(); + Ok(vec![0; size / core::mem::size_of::()]) + } + + fn reserve_address_space() -> Result { + Ok(AddressSpace { + native_code_origin: NATIVE_CODE_ORIGIN, + }) + } + + fn prepare_program( + _global: &Self::GlobalState, + init: SandboxInit, + _address_space: Self::AddressSpace, + ) -> Result { + let cfg = init.guest_init.memory_map().map_err(Error::from)?; + + // The jump table is a separate byte blob; the compiler places it right after + // the code, reached via a PC-relative `adr` (see compiler/aarch64.rs). + let jump_table = as_bytes(init.jump_table.as_ref()).to_vec(); + + let mut memory_map = Vec::new(); + let mut push_region = |address: u32, data: &[u8], virtual_size: u32, is_writable: bool| { + if virtual_size == 0 { + return; + } + let physical = core::cmp::min(data.len() as u32, virtual_size); + if physical > 0 { + memory_map.push(ProgramMap { + address, + length: physical, + is_writable, + data: Some(Arc::from(&data[..physical as usize])), + }); + } + if virtual_size > physical { + memory_map.push(ProgramMap { + address: address + physical, + length: virtual_size - physical, + is_writable, + data: None, + }); + } + }; + + push_region(cfg.ro_data_address(), init.guest_init.ro_data, cfg.ro_data_size(), false); + push_region(cfg.rw_data_address(), init.guest_init.rw_data, cfg.rw_data_size(), true); + + // Reserve a (transient) entry for the heap right after rw-data. + let heap_map_index = memory_map.len(); + memory_map.push(ProgramMap { + address: cfg.rw_data_range().end, + length: 0, + is_writable: true, + data: None, + }); + + if cfg.stack_size() > 0 { + memory_map.push(ProgramMap { + address: cfg.stack_address_low(), + length: cfg.stack_size(), + is_writable: true, + data: None, + }); + } + if cfg.aux_data_size() > 0 { + memory_map.push(ProgramMap { + address: cfg.aux_data_address(), + length: cfg.aux_data_size(), + is_writable: true, + data: None, + }); + } + + Ok(SandboxProgram(Arc::new(SandboxProgramInner { + code: init.code.to_vec(), + jump_table, + sysenter_address: init.sysenter_address, + sysreturn_address: init.sysreturn_address, + memory_map, + heap_map_index, + }))) + } + + fn spawn(_global: &Self::GlobalState, _config: &Self::Config, _outer_instance: Option<&Self>) -> Result, Self::Error> { + let vm = Vm::new()?; + Ok(Box::new(Sandbox { + vm, + module: None, + program: None, + gas_metering: None, + regs: [0; REG_COUNT], + gas: 0, + program_counter: None, + is_program_counter_valid: false, + next_program_counter: None, + next_program_counter_changed: false, + charge_gas_on_entry: true, + page_size: GUEST_PAGE_SIZE as u32, + accessible_aux_size: 0, + aux_data_address: 0, + aux_data_full_length: 0, + heap_base: 0, + heap_top: 0, + dynamic_paging: false, + page_prot: vec![PROT_READ_WRITE; NUM_PAGES], + })) + } + + fn load_module(&mut self, _global: &Self::GlobalState, module: &Module) -> Result<(), Self::Error> { + #[cfg(target_os = "macos")] + { + let compiled = Self::downcast_module(module); + let program = compiled.sandbox_program.clone(); + + // Assemble the code + inline jump table blob and map it RX. + let mut blob = program.0.code.clone(); + // The compiler pads code to a page and defines the jump-table label at the end; + // append the jump-table bytes so the PC-relative `adr` lands on real data. + blob.extend_from_slice(&program.0.jump_table); + self.vm.load_code(&blob)?; + + // The return-to-host jump-table slot lives far into the (sparse) virtual jump + // table, at `jump_table_base + RETURN_TO_HOST*8`. `jump_table_base` is the + // padded end of the code (where the codegen's `adr` points). Map that single + // page and store the sysreturn stub address so a `ret` reaches the host. + let jump_table_base = NATIVE_CODE_ORIGIN + program.0.code.len() as u64; + let slot_va = jump_table_base + (u64::from(VM_ADDR_RETURN_TO_HOST) << 3); + debug_assert_eq!(slot_va, jump_table_base + (VM_ADDR_JUMP_TABLE_RETURN_TO_HOST - VM_ADDR_JUMP_TABLE)); + self.vm.map_return_slot(slot_va, program.0.sysreturn_address)?; + + self.dynamic_paging = module.is_dynamic_paging(); + // Start with the whole window inaccessible so out-of-bounds guest accesses fault + // (static -> Trap, dynamic -> Segfault), mirroring the generic sandbox. + self.vm.clear_window(); + self.page_prot.fill(PROT_NONE); + if !self.dynamic_paging { + // Static paging: materialize the memory map into guest RAM and enable it. + for map in &program.0.memory_map { + if map.length == 0 { + continue; + } + if let Some(data) = &map.data { + self.vm + .ram_slice_mut(map.address, data.len()) + .ok_or_else(|| Error::from("memory map region out of range"))? + .copy_from_slice(data); + } + let ap = if map.is_writable { PTE_AP_RW_EL1 } else { PTE_AP_RO_EL1 }; + self.vm.set_page_protection(map.address, map.length, ap); + let state = if map.is_writable { PROT_READ_WRITE } else { PROT_READ }; + mark_pages(&mut self.page_prot, map.address, map.length, state); + } + } + // Dynamic paging leaves everything faulted; the host pages it in on demand. + + // Initialize the guest VmCtx. + let mmap = module.memory_map(); + // SAFETY: `vmctx_ptr` points at the mapped, zero-initialized VmCtx page. + unsafe { + core::ptr::write(self.vm.vmctx_ptr(), VmCtx::new()); + let vmctx = &mut *self.vm.vmctx_ptr(); + vmctx.heap_info.heap_top = u64::from(mmap.heap_base()); + vmctx.heap_info.heap_threshold = u64::from(mmap.rw_data_range().end); + vmctx.heap_base = mmap.heap_base(); + vmctx.heap_initial_threshold = mmap.rw_data_range().end; + vmctx.heap_max_size = mmap.max_heap_size(); + vmctx.heap_map_index = program.0.heap_map_index; + vmctx.page_size = mmap.page_size(); + } + + self.heap_base = mmap.heap_base(); + self.heap_top = mmap.heap_base(); + if mmap.page_size() as usize % GUEST_PAGE_SIZE != 0 { + return Err(alloc::format!( + "module page size ({}) must be a multiple of the guest granule ({GUEST_PAGE_SIZE})", + mmap.page_size() + ) + .into()); + } + self.page_size = mmap.page_size(); + self.aux_data_address = mmap.aux_data_address(); + self.aux_data_full_length = mmap.aux_data_size(); + // Fully accessible until the host narrows it. + self.accessible_aux_size = self.aux_data_full_length; + self.gas_metering = module.gas_metering(); + self.program = Some(program); + self.module = Some(module.clone()); + Ok(()) + } + #[cfg(not(target_os = "macos"))] + { + let _ = module; + todo!("hypervisor sandbox: load_module needs the KVM backend") + } + } + + fn recycle(_sandbox: Box, _global: &Self::GlobalState) -> Result<(), Self::Error> { + // Dropping the box tears down the vCPU + VM. + Ok(()) + } + + /// Guest addresses of the `hvc` trampoline stubs (see `install_stubs`). + fn address_table() -> AddressTable { + AddressTable { + syscall_hostcall: STUBS_IPA + u64::from(HVC_HOSTCALL - 1) * STUB_STRIDE, + syscall_trap: STUBS_IPA + u64::from(HVC_TRAP - 1) * STUB_STRIDE, + syscall_return: STUBS_IPA + u64::from(HVC_RETURN - 1) * STUB_STRIDE, + syscall_step: STUBS_IPA + u64::from(HVC_STEP - 1) * STUB_STRIDE, + syscall_sbrk: STUBS_IPA + u64::from(HVC_SBRK - 1) * STUB_STRIDE, + syscall_not_enough_gas: STUBS_IPA + u64::from(HVC_NOT_ENOUGH_GAS - 1) * STUB_STRIDE, + } + } + + /// Offsets into the guest `VmCtx`; must match the shared AArch64 codegen. + fn offset_table() -> OffsetTable { + OffsetTable { + arg: get_field_offset!(VmCtx::new(), |base| base.arg.as_ptr()), + gas: get_field_offset!(VmCtx::new(), |base| base.gas.as_ptr()), + heap_info: get_field_offset!(VmCtx::new(), |base| &base.heap_info), + next_native_program_counter: get_field_offset!(VmCtx::new(), |base| base.next_native_program_counter.as_ptr()), + next_program_counter: get_field_offset!(VmCtx::new(), |base| base.next_program_counter.as_ptr()), + program_counter: get_field_offset!(VmCtx::new(), |base| base.program_counter.as_ptr()), + regs: get_field_offset!(VmCtx::new(), |base| &base.regs), + memset_continuation: get_field_offset!(VmCtx::new(), |base| base.memset_continuation.as_ptr()), + futex: usize::MAX, + } + } + + fn idle_worker_pids(_global: &Self::GlobalState) -> Vec { + Vec::new() + } + + // First cut: enter via the `sysenter` stub, run the vCPU, and translate the + // `hvc` / abort exits into `InterruptKind`, mirroring generic.rs:1880-1896. + // NOTE(hypervisor): unverified — needs the hypervisor entitlement to execute, and + // full correctness also depends on codegen assumptions documented in the report. + fn run(&mut self) -> Result { + #[cfg(target_os = "macos")] + { + let module = self.module.as_ref().ok_or_else(|| Error::from("no module loaded"))?.clone(); + let compiled = Self::downcast_module(&module); + + if self.next_program_counter_changed { + let pc = self + .next_program_counter + .ok_or_else(|| Error::from("next program counter not set"))?; + let Some(address) = compiled.lookup_native_code_address(pc) else { + // `program_counter()` reads this back out of the VmCtx. + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + (*self.vm.vmctx_ptr()) + .program_counter + .store(pc.0, core::sync::atomic::Ordering::Relaxed); + } + self.program_counter = Some(pc); + self.is_program_counter_valid = true; + return Ok(InterruptKind::Trap); + }; + if self.charge_gas_on_entry { + match crate::sandbox::charge_gas_on_entry(&module, pc, address, compiled, self.gas) { + Some(Ok(new_gas)) => self.gas = new_gas, + Some(Err(())) => return Ok(InterruptKind::NotEnoughGas), + None => {} + } + } + self.next_program_counter_changed = false; + self.charge_gas_on_entry = false; + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + let vmctx = &mut *self.vm.vmctx_ptr(); + vmctx.next_program_counter.store(pc.0, core::sync::atomic::Ordering::Relaxed); + vmctx + .next_native_program_counter + .store(address, core::sync::atomic::Ordering::Relaxed); + } + self.next_program_counter = None; + } + + let sysenter = self.program.as_ref().unwrap().0.sysenter_address; + + // Push host state into the guest VmCtx before entering. + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + let vmctx = &mut *self.vm.vmctx_ptr(); + vmctx.exit_reason = HvExitReason::None; + vmctx.gas.store(self.gas, core::sync::atomic::Ordering::Relaxed); + for (i, value) in self.regs.iter().enumerate() { + vmctx.regs.0[i] = *value; + } + } + + self.is_program_counter_valid = true; + let interrupt = self.execute(sysenter)?; + + // Pull guest state back out. + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + let vmctx = &*self.vm.vmctx_ptr(); + self.gas = vmctx.gas.load(core::sync::atomic::Ordering::Relaxed); + for (i, value) in self.regs.iter_mut().enumerate() { + *value = vmctx.regs.0[i]; + } + } + + // Async metering doesn't trap in-guest; the counter is only checked on the way out. + if self.gas_metering == Some(GasMeteringKind::Async) && self.gas < 0 { + self.is_program_counter_valid = false; + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + (*self.vm.vmctx_ptr()) + .next_native_program_counter + .store(0, core::sync::atomic::Ordering::Relaxed); + } + return Ok(InterruptKind::NotEnoughGas); + } + + Ok(interrupt) + } + #[cfg(not(target_os = "macos"))] + { + todo!("hypervisor sandbox: run needs the KVM backend") + } + } + + fn reg(&self, reg: Reg) -> RegValue { + self.regs[reg as usize] + } + + fn set_reg(&mut self, reg: Reg, value: RegValue) { + self.regs[reg as usize] = value; + } + + fn gas(&self) -> Gas { + self.gas + } + + fn set_gas(&mut self, gas: Gas) { + self.gas = gas; + } + + fn program_counter(&self) -> Option { + if !self.is_program_counter_valid { + return None; + } + #[cfg(target_os = "macos")] + { + // Written by the guest stubs and the fault paths. + // SAFETY: the VmCtx page is mapped and initialized. + let vmctx = unsafe { &*self.vm.vmctx_ptr() }; + Some(ProgramCounter(vmctx.program_counter.load(core::sync::atomic::Ordering::Relaxed))) + } + #[cfg(not(target_os = "macos"))] + { + self.program_counter + } + } + + fn next_program_counter(&self) -> Option { + if self.next_program_counter.is_some() { + return self.next_program_counter; + } + // After an in-guest exit (e.g. ecall) the resume PC lives in the VmCtx. + #[cfg(target_os = "macos")] + { + use core::sync::atomic::Ordering; + // SAFETY: the VmCtx page is mapped and initialized once a module is loaded. + let vmctx = unsafe { &*self.vm.vmctx_ptr() }; + if vmctx.next_native_program_counter.load(Ordering::Relaxed) == 0 { + None + } else { + Some(ProgramCounter(vmctx.next_program_counter.load(Ordering::Relaxed))) + } + } + #[cfg(not(target_os = "macos"))] + { + None + } + } + + fn next_native_program_counter(&self) -> Option { + #[cfg(target_os = "macos")] + { + // SAFETY: the VmCtx page is mapped and zeroed before a module is loaded. + let vmctx = unsafe { &*self.vm.vmctx_ptr() }; + match vmctx.next_native_program_counter.load(core::sync::atomic::Ordering::Relaxed) { + 0 => None, + address => Some(address as usize), + } + } + #[cfg(not(target_os = "macos"))] + { + None + } + } + + fn set_next_program_counter(&mut self, pc: ProgramCounter) { + self.is_program_counter_valid = false; + self.next_program_counter = Some(pc); + self.next_program_counter_changed = true; + self.charge_gas_on_entry = true; + } + + fn accessible_aux_size(&self) -> u32 { + self.accessible_aux_size + } + + fn set_accessible_aux_size(&mut self, size: u32) -> Result<(), Self::Error> { + if self.aux_data_address == 0 || self.aux_data_full_length == 0 { + return Err(Error::from("aux data address or length is zero")); + } + if size > self.aux_data_full_length { + return Err(Error::from("size exceeds the full length of aux data")); + } + self.accessible_aux_size = size; + Ok(()) + } + + fn is_memory_accessible(&self, address: u32, size: u32, minimum_protection: MemoryProtection) -> bool { + if size == 0 { + return true; + } + if !bounds_ok(address, size) { + return false; + } + let required = match minimum_protection { + MemoryProtection::Read => PROT_READ, + MemoryProtection::ReadWrite => PROT_READ_WRITE, + }; + self.accessible_for(address, size, required) + } + + fn reset_memory(&mut self) -> Result<(), Self::Error> { + #[cfg(target_os = "macos")] + { + if self.module.is_none() { + return Err("no module loaded into the sandbox".into()); + } + if self.dynamic_paging { + return self.free_pages(0x10000, 0xffff_0000); + } + + // Static paging: the guest can only have written where the map made it writable, so + // restore those regions instead of scrubbing the whole 4 GiB window. + let program = self.program.clone().ok_or_else(|| Error::from("no program loaded"))?; + + // Anything `sbrk` grew into lies outside the map; revoke and clear it. + // SAFETY: the VmCtx page is mapped and initialized. + let (threshold, initial) = unsafe { + let vmctx = &*self.vm.vmctx_ptr(); + (vmctx.heap_info.heap_threshold as u32, vmctx.heap_initial_threshold) + }; + if threshold > initial { + let length = threshold - initial; + if let Some(slice) = self.vm.ram_slice_mut(initial, length as usize) { + slice.fill(0); + } + self.vm.invalidate_pages(initial, length); + mark_pages(&mut self.page_prot, initial, length, PROT_NONE); + } + + for map in &program.0.memory_map { + if !map.is_writable || map.length == 0 { + continue; + } + let Some(slice) = self.vm.ram_slice_mut(map.address, map.length as usize) else { + return Err("memory map region out of range".into()); + }; + match &map.data { + Some(data) => slice.copy_from_slice(data), + None => slice.fill(0), + } + } + + self.heap_top = self.heap_base; + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + let vmctx = &mut *self.vm.vmctx_ptr(); + vmctx.heap_info.heap_top = u64::from(self.heap_base); + vmctx.heap_info.heap_threshold = u64::from(initial); + } + Ok(()) + } + #[cfg(not(target_os = "macos"))] + { + todo!("hypervisor sandbox: reset_memory needs the KVM backend") + } + } + + fn read_memory_into<'slice>(&self, address: u32, slice: &'slice mut [MaybeUninit]) -> Result<&'slice mut [u8], MemoryAccessError> { + if slice.is_empty() { + // SAFETY: empty slice; nothing is read. + return Ok(unsafe { core::slice::from_raw_parts_mut(slice.as_mut_ptr().cast(), 0) }); + } + if !bounds_ok(address, slice.len() as u32) || !self.accessible_for(address, slice.len() as u32, PROT_READ) { + return Err(MemoryAccessError::OutOfRangeAccess { + address, + length: slice.len() as u64, + }); + } + #[cfg(target_os = "macos")] + { + let src = self.vm.ram_slice(address, slice.len()).ok_or(MemoryAccessError::OutOfRangeAccess { + address, + length: slice.len() as u64, + })?; + for (dst, byte) in slice.iter_mut().zip(src.iter()) { + dst.write(*byte); + } + // SAFETY: every element was just initialized. + Ok(unsafe { core::slice::from_raw_parts_mut(slice.as_mut_ptr().cast(), slice.len()) }) + } + #[cfg(not(target_os = "macos"))] + { + todo!("hypervisor sandbox: guest RAM access needs the KVM backend") + } + } + + fn write_memory(&mut self, address: u32, data: &[u8]) -> Result<(), MemoryAccessError> { + if data.is_empty() { + return Ok(()); + } + if !bounds_ok(address, data.len() as u32) || !self.accessible_for(address, data.len() as u32, PROT_READ_WRITE) { + return Err(MemoryAccessError::OutOfRangeAccess { + address, + length: data.len() as u64, + }); + } + #[cfg(target_os = "macos")] + { + let dst = self + .vm + .ram_slice_mut(address, data.len()) + .ok_or(MemoryAccessError::OutOfRangeAccess { + address, + length: data.len() as u64, + })?; + dst.copy_from_slice(data); + Ok(()) + } + #[cfg(not(target_os = "macos"))] + { + todo!("hypervisor sandbox: guest RAM access needs the KVM backend") + } + } + + fn zero_memory(&mut self, address: u32, length: u32, memory_protection: Option) -> Result<(), MemoryAccessError> { + if length == 0 { + return Ok(()); + } + if !bounds_ok(address, length) { + return Err(MemoryAccessError::OutOfRangeAccess { + address, + length: u64::from(length), + }); + } + // `Some(protection)` = dynamic-paging page-in: make the pages present with that + // protection, then zero them. `None` requires the pages already be writable. + if let Some(protection) = memory_protection { + #[cfg(target_os = "macos")] + { + let (ap, state) = match protection { + MemoryProtection::Read => (PTE_AP_RO_EL1, PROT_READ), + MemoryProtection::ReadWrite => (PTE_AP_RW_EL1, PROT_READ_WRITE), + }; + if let Some(dst) = self.vm.ram_slice_mut(address, length as usize) { + dst.fill(0); + } + self.vm.set_page_protection(address, length, ap); + mark_pages(&mut self.page_prot, address, length, state); + return Ok(()); + } + #[cfg(not(target_os = "macos"))] + { + let _ = protection; + todo!("hypervisor sandbox: guest RAM access needs the KVM backend") + } + } + if !self.accessible_for(address, length, PROT_READ_WRITE) { + return Err(MemoryAccessError::OutOfRangeAccess { + address, + length: u64::from(length), + }); + } + #[cfg(target_os = "macos")] + { + let dst = self + .vm + .ram_slice_mut(address, length as usize) + .ok_or(MemoryAccessError::OutOfRangeAccess { + address, + length: u64::from(length), + })?; + dst.fill(0); + Ok(()) + } + #[cfg(not(target_os = "macos"))] + { + todo!("hypervisor sandbox: guest RAM access needs the KVM backend") + } + } + + fn change_memory_protection(&mut self, address: u32, length: u32, protection: MemoryProtection) -> Result<(), MemoryAccessError> { + if length == 0 { + return Ok(()); + } + if !bounds_ok(address, length) { + return Err(MemoryAccessError::OutOfRangeAccess { + address, + length: u64::from(length), + }); + } + #[cfg(target_os = "macos")] + { + // Only present pages can be re-protected. + if !self.accessible_for(address, length, PROT_READ) { + return Err(MemoryAccessError::OutOfRangeAccess { + address, + length: u64::from(length), + }); + } + let (ap, state) = match protection { + MemoryProtection::Read => (PTE_AP_RO_EL1, PROT_READ), + MemoryProtection::ReadWrite => (PTE_AP_RW_EL1, PROT_READ_WRITE), + }; + self.vm.set_page_protection(address, length, ap); + mark_pages(&mut self.page_prot, address, length, state); + Ok(()) + } + #[cfg(not(target_os = "macos"))] + { + let _ = protection; + todo!("hypervisor sandbox: page protection needs the KVM backend") + } + } + + fn free_pages(&mut self, address: u32, length: u32) -> Result<(), Self::Error> { + if length == 0 { + return Ok(()); + } + if !bounds_ok(address, length) { + return Err("free_pages: out of range".into()); + } + #[cfg(target_os = "macos")] + { + // Walk the page tables only for pages that are actually present: freeing the whole + // window is the common case (`reset_memory`) and is nearly always sparse. + let first = address as usize / GUEST_PAGE_SIZE; + let last = (address as usize + length as usize - 1) / GUEST_PAGE_SIZE; + let mut page = first; + while page <= last { + if self.page_prot[page] == PROT_NONE { + page += 1; + continue; + } + let start = page; + // Cap a run at 1 GiB so the byte length always fits a `u32`. + let limit = core::cmp::min(last, start + (1 << 18) - 1); + while page <= limit && self.page_prot[page] != PROT_NONE { + page += 1; + } + let va = (start * GUEST_PAGE_SIZE) as u32; + let len = ((page - start) * GUEST_PAGE_SIZE) as u32; + self.vm.invalidate_pages(va, len); + self.page_prot[start..page].fill(PROT_NONE); + } + Ok(()) + } + #[cfg(not(target_os = "macos"))] + { + todo!("hypervisor sandbox: free_pages needs the KVM backend") + } + } + + fn heap_size(&self) -> u32 { + self.heap_top - self.heap_base + } + + // First cut: grow the heap top; the pages already exist in the identity mapping. + // NOTE(hypervisor): a full implementation would map fresh 4K guest pages on demand. + fn sbrk(&mut self, size: u32) -> Result, Self::Error> { + let module = self.module.as_ref().ok_or_else(|| Error::from("no module loaded"))?; + let max_heap_size = module.memory_map().max_heap_size(); + let new_top = self.heap_top.checked_add(size).ok_or_else(|| Error::from("sbrk overflow"))?; + if new_top - self.heap_base > max_heap_size { + return Ok(None); + } + self.heap_top = new_top; + #[cfg(target_os = "macos")] + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + (*self.vm.vmctx_ptr()).heap_info.heap_top = u64::from(new_top); + } + Ok(Some(new_top)) + } + + fn pid(&self) -> Option { + None + } +} + +impl Sandbox { + /// Host memory-access permission check. Only restricts under dynamic paging (where pages + /// must be explicitly mapped in); static paging keeps the whole window accessible. + fn accessible_for(&self, address: u32, length: u32, required: u8) -> bool { + if length == 0 { + return true; + } + let Some(address_end) = address.checked_add(length) else { + return false; + }; + + // Aux data is bounded by what the host made accessible, not the region's full size. + if address >= self.aux_data_address + && self.aux_data_full_length != 0 + && address_end < self.aux_data_address + self.aux_data_full_length + { + return address_end <= self.aux_data_address + self.accessible_aux_size; + } + + // `page_prot` tracks what is mapped under either paging mode, including `sbrk` growth. + let first = address as usize / GUEST_PAGE_SIZE; + let last = (address_end as usize - 1) / GUEST_PAGE_SIZE; + self.page_prot[first..=last].iter().all(|&p| p >= required) + } + + /// Drive the vCPU from `entry` (the sysenter stub) until it returns to the host, + /// translating exits to `InterruptKind`. First cut; see the `run` NOTE. + #[cfg(target_os = "macos")] + fn execute(&mut self, entry: u64) -> Result { + use core::sync::atomic::Ordering; + + self.apply_aux_data_protection(); + + // Any page-table edits since the last entry (paging in, protection changes, `free_pages`, + // aux data) must be visible to the guest before it runs again. + self.vm.flush_tlb()?; + + // The guest expects x14 = memory base and x13 = tmp_reg on entry (set by run). + let tmp_reg = { + // SAFETY: VmCtx is mapped. + let vmctx = unsafe { &*self.vm.vmctx_ptr() }; + vmctx.tmp_reg.load(Ordering::Relaxed) + }; + self.vm.set_reg(14, GUEST_MEM_BASE)?; // AUX_TMP_REG / memory base + self.vm.set_reg(13, tmp_reg)?; // TMP_REG + self.vm.set_reg(hv::HV_REG_PC, entry)?; + + loop { + match self.vm.run_raw()? { + VmExit::HostCall { imm } => match imm { + HVC_HOSTCALL => { + // SAFETY: VmCtx is mapped. + let arg = unsafe { (*self.vm.vmctx_ptr()).arg.load(Ordering::Relaxed) }; + return Ok(InterruptKind::Ecalli(arg)); + } + HVC_TRAP => return Ok(InterruptKind::Trap), + HVC_RETURN => { + self.is_program_counter_valid = false; + return Ok(InterruptKind::Finished); + } + HVC_STEP => return Ok(InterruptKind::Step), + HVC_NOT_ENOUGH_GAS => return Ok(InterruptKind::NotEnoughGas), + HVC_SBRK => { + // sbrk is resumable: compute the new heap top and hand it back in x0. + // The vCPU PC already points past the `hvc` (at the stub's `ret`), so + // just continue the loop to run it. + let pending = self.vm.get_reg(hv::HV_REG_X0)?; + let result = self.sbrk_pending(pending); + self.vm.set_reg(hv::HV_REG_X0, u64::from(result))?; + } + HVC_FAULT => { + // The guest EL1 vector reports a stage-1 abort; details are in EL1 sysregs. + let far = self.vm.get_sys_reg(hv::HV_SYS_REG_FAR_EL1)?; + let esr = self.vm.get_sys_reg(hv::HV_SYS_REG_ESR_EL1)?; + let elr = self.vm.get_sys_reg(hv::HV_SYS_REG_ELR_EL1)?; + return self.handle_fault(far, esr, elr); + } + _ => return Ok(InterruptKind::Trap), + }, + // A stage-2 abort (from hv itself) surfaces directly; PC is the faulting instr. + VmExit::MemoryFault { address, esr } => { + let pc = self.vm.get_reg(hv::HV_REG_PC)?; + return self.handle_fault(address, esr, pc); + } + VmExit::Trap { .. } | VmExit::Exception { .. } | VmExit::Unknown => return Ok(InterruptKind::Trap), + VmExit::Canceled | VmExit::VTimer => return Err("vCPU run was canceled".into()), + } + } + } + + /// Translate a guest fault (`FAR`/`ESR`, faulting instr at `elr`) into an `InterruptKind`. + /// Mirrors generic: a dynamic-paging abort above the null guard becomes a `Segfault` whose + /// faulting instruction can be re-executed on the next `run` (registers snapshotted into the + /// VmCtx, resume address recomputed); anything else is a `Trap`. + #[cfg(target_os = "macos")] + fn handle_fault(&mut self, far: u64, esr: u64, elr: u64) -> Result { + use core::sync::atomic::Ordering; + + let ec = (esr >> 26) & 0x3f; + // EC 0x00 ("unknown") is an undefined instruction: either the sync gas-metering `udf` or + // executing garbage. Only the former has a negative gas counter. + if ec == 0x00 { + if let Some(interrupt) = self.handle_gas_trap(elr)? { + return Ok(interrupt); + } + return self.trap_at(elr); + } + // Data abort (0x24/0x25) or instruction abort (0x20/0x21). + if !matches!(ec, 0x20 | 0x21 | 0x24 | 0x25) { + return self.trap_at(elr); + } + let guest_addr = far.wrapping_sub(GUEST_MEM_BASE); + let page_address = (guest_addr as u32) & !(self.page_size - 1); + if !(self.dynamic_paging && guest_addr < RAM_SIZE as u64 && page_address >= 0x10000) { + return self.trap_at(elr); + } + // Like generic: `is_write_protected` means the page exists but is read-only (vs absent). + let page_idx = page_address as usize / GUEST_PAGE_SIZE; + let is_write_protected = self.page_prot[page_idx] == PROT_READ; + + let module = self.module.clone().ok_or_else(|| Error::from("fault: no module"))?; + let compiled = ::downcast_module(&module); + + self.snapshot_regs()?; + + // Map the faulting native PC to a guest program counter and the native address to resume + // at (start of that instruction, so re-running retries the access). + let offset = elr.wrapping_sub(compiled.native_code_origin); + let pc = compiled + .program_counter_by_native_code_offset(offset, false) + .ok_or_else(|| Error::from("fault: no program counter for faulting address"))?; + let resume = compiled.resume_native_address_for_pagefault(pc, elr, self.gas_metering); + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + let vmctx = &mut *self.vm.vmctx_ptr(); + vmctx.program_counter.store(pc.0, Ordering::Relaxed); + vmctx.next_program_counter.store(pc.0, Ordering::Relaxed); + vmctx.next_native_program_counter.store(resume, Ordering::Relaxed); + } + self.program_counter = Some(pc); + self.is_program_counter_valid = true; + self.next_program_counter = None; + self.next_program_counter_changed = false; + self.charge_gas_on_entry = false; + + Ok(InterruptKind::Segfault(Segfault { + page_address, + page_size: self.page_size, + is_write_protected, + })) + } + + /// Guest may read only the first `accessible_aux_size` bytes of aux data; the rest faults. Like + /// generic's `set_aux_data_permission_for_guest`; the host is unaffected (it uses its own map). + #[cfg(target_os = "macos")] + fn apply_aux_data_protection(&mut self) { + if self.aux_data_full_length == 0 || self.dynamic_paging { + return; + } + self.vm.invalidate_pages(self.aux_data_address, self.aux_data_full_length); + if self.accessible_aux_size > 0 { + self.vm + .set_page_protection(self.aux_data_address, self.accessible_aux_size, PTE_AP_RO_EL1); + } + } + + /// Copy the live guest registers into the VmCtx so the host can read/edit them and `sysenter` + /// restores them. The `hvc` trampolines do this themselves; the EL1 vector does not. + #[cfg(target_os = "macos")] + fn snapshot_regs(&mut self) -> Result<(), Error> { + for reg in Reg::ALL { + let value = self.vm.get_reg(to_native_reg(reg) as u32)?; + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { (*self.vm.vmctx_ptr()).regs.0[reg as usize] = value }; + } + Ok(()) + } + + /// Plain `Trap`: the EL1 vector saved nothing, so recover regs and the PC from `elr`, else from + /// the jump site the indirect-jump codegen stashed. + #[cfg(target_os = "macos")] + fn trap_at(&mut self, elr: u64) -> Result { + use core::sync::atomic::Ordering; + + self.snapshot_regs()?; + let Some(module) = self.module.clone() else { + return Ok(InterruptKind::Trap); + }; + let compiled = ::downcast_module(&module); + // SAFETY: the VmCtx page is mapped and initialized. + let stashed = unsafe { (*self.vm.vmctx_ptr()).next_native_program_counter.load(Ordering::Relaxed) }; + let code_len = compiled.machine_code().len() as u64; + + for address in [elr, stashed] { + let Some(offset) = address.checked_sub(compiled.native_code_origin) else { + continue; + }; + if offset >= code_len { + continue; + } + let Some(pc) = compiled.program_counter_by_native_code_offset(offset, false) else { + continue; + }; + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + let vmctx = &mut *self.vm.vmctx_ptr(); + vmctx.program_counter.store(pc.0, Ordering::Relaxed); + vmctx.next_native_program_counter.store(0, Ordering::Relaxed); + } + self.program_counter = Some(pc); + self.is_program_counter_valid = true; + return Ok(InterruptKind::Trap); + } + + Ok(InterruptKind::Trap) + } + + /// The sync metering stub's `udf` fired: refund the charged gas, snapshot registers and re-enter + /// at the stub, as generic's `handle_guest_signal` does. `None` if it wasn't a gas trap. + #[cfg(target_os = "macos")] + fn handle_gas_trap(&mut self, elr: u64) -> Result, Error> { + use core::sync::atomic::Ordering; + + // SAFETY: the VmCtx page is mapped and initialized. + let gas = unsafe { (*self.vm.vmctx_ptr()).gas.load(Ordering::Relaxed) }; + if self.gas_metering.is_none() || gas >= 0 { + return Ok(None); + } + + let module = self.module.clone().ok_or_else(|| Error::from("gas trap: no module"))?; + let compiled = ::downcast_module(&module); + // The `udf` sits a fixed distance into the stub; back up to its first instruction so + // re-entering recharges the refunded gas. + let machine_code_offset = elr.wrapping_sub(compiled.native_code_origin); + let Some(offset) = machine_code_offset.checked_sub(crate::compiler::GAS_METERING_TRAP_OFFSET) else { + return Ok(None); + }; + let Some(pc) = compiled.program_counter_by_native_code_offset(offset, false) else { + return Ok(None); + }; + + let cost = crate::compiler::extract_gas_cost::(compiled.machine_code(), offset as usize); + self.snapshot_regs()?; + // SAFETY: the VmCtx page is mapped and initialized. + unsafe { + let vmctx = &mut *self.vm.vmctx_ptr(); + vmctx.gas.fetch_add(i64::from(cost), Ordering::Relaxed); + vmctx.program_counter.store(pc.0, Ordering::Relaxed); + vmctx.next_program_counter.store(pc.0, Ordering::Relaxed); + vmctx + .next_native_program_counter + .store(compiled.native_code_origin + offset, Ordering::Relaxed); + } + self.program_counter = Some(pc); + self.is_program_counter_valid = true; + self.next_program_counter = None; + self.next_program_counter_changed = false; + self.charge_gas_on_entry = false; + + Ok(Some(InterruptKind::NotEnoughGas)) + } + + /// Heap-growth helper shared by the `sbrk` hvc path: returns the new heap top or 0. + /// Mirrors generic's `sbrk`: enable the newly-crossed 4K pages RW and bump the + /// heap top/threshold in the VmCtx. + #[cfg(target_os = "macos")] + fn sbrk_pending(&mut self, pending_top: u64) -> u32 { + let Some(module) = self.module.as_ref() else { return 0 }; + let mmap = module.memory_map(); + let page = u64::from(mmap.page_size()); + if pending_top > u64::from(mmap.heap_base()) + u64::from(mmap.max_heap_size()) { + return 0; + } + // SAFETY: VmCtx is mapped. + let old_top = unsafe { (*self.vm.vmctx_ptr()).heap_info.heap_top }; + let start = old_top.next_multiple_of(page); + let end = pending_top.next_multiple_of(page); + if end > start { + let len = (end - start) as u32; + self.vm.set_page_protection(start as u32, len, PTE_AP_RW_EL1); + mark_pages(&mut self.page_prot, start as u32, len, PROT_READ_WRITE); + } + self.heap_top = pending_top as u32; + // SAFETY: VmCtx is mapped. + unsafe { + let vmctx = &mut *self.vm.vmctx_ptr(); + vmctx.heap_info.heap_top = pending_top; + vmctx.heap_info.heap_threshold = end; + } + pending_top as u32 + } +} + +/// Mark `[address, address + length)` (PolkaVM addresses) with a protection state. +fn mark_pages(page_prot: &mut [u8], address: u32, length: u32, state: u8) { + let first = address as usize / GUEST_PAGE_SIZE; + let last = (address as usize + length as usize - 1) / GUEST_PAGE_SIZE; + page_prot[first..=last].fill(state); +} + +/// Reinterpret a `&[usize]` (the jump table) as bytes. +fn as_bytes(slice: &[usize]) -> &[u8] { + // SAFETY: `u8` has no alignment requirement and any bit pattern is valid. + unsafe { core::slice::from_raw_parts(slice.as_ptr().cast(), core::mem::size_of_val(slice)) } +} diff --git a/crates/polkavm/src/tests.rs b/crates/polkavm/src/tests.rs index 514ad9680..8839a51d7 100644 --- a/crates/polkavm/src/tests.rs +++ b/crates/polkavm/src/tests.rs @@ -117,9 +117,31 @@ fn get_test_program(kind: TestProgram, is_64_bit: bool) -> &'static [u8] { } } +/// Opt-in: needs `POLKAVM_TEST_HYPERVISOR` and an entitled binary. +#[cfg(all(target_os = "macos", target_arch = "aarch64", feature = "hypervisor-sandbox"))] +fn hypervisor_matrix_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| std::env::var("POLKAVM_TEST_HYPERVISOR").is_ok() && crate::sandbox::hypervisor::Vm::new().is_ok()) +} + +/// One VM per process, vCPU bound to its creating thread: tests needing several live instances, or +/// one on another thread, can't run in that lane. +fn skip_if_single_instance_sandbox(config: &Config) -> bool { + #[cfg(all(target_os = "macos", target_arch = "aarch64", feature = "hypervisor-sandbox"))] + if config.sandbox() == Some(crate::SandboxKind::Hypervisor) { + return true; + } + let _ = config; + false +} + fn get_native_page_size() -> usize { if_compiler_is_supported! { - { crate::sandbox::get_native_page_size() } else { 4096 } + { + // Normally primed by `Engine::new`; tests may ask before building an engine. + crate::sandbox::init_native_page_size(); + crate::sandbox::get_native_page_size() + } else { 4096 } } } @@ -181,6 +203,21 @@ macro_rules! run_tests_on_isa { config.set_crosscheck(true); $test_name(config, $isa); } + + // The only 4K-granule lane on a 16K host; opt-in because it needs an entitled + // binary and serializes on the single per-process VM. + #[cfg(all(target_os = "macos", target_arch = "aarch64", feature = "hypervisor-sandbox"))] + #[test] + fn []() { + if !crate::tests::hypervisor_matrix_enabled() { + return; + } + let mut config = crate::Config::default(); + config.set_backend(Some(crate::BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + config.set_allow_experimental(true); + $test_name(config, $isa); + } } )+ } @@ -380,6 +417,420 @@ fn basic_test(config: Config, isa: InstructionSetKind) { assert_eq!(result, 111); } +// End-to-end tests for the AArch64 recompiler backend, exercising it through the generic sandbox. +// They run only where that sandbox is available (notably aarch64-apple-darwin) and a real CPU executes +// the generated code. +#[cfg(feature = "generic-sandbox")] +mod aarch64_backend { + use super::*; + use crate::{RawInstance, RegValue}; + use polkavm_common::program::Instruction; + + /// Selects the sandbox + guest page size. With `POLKAVM_TEST_HYPERVISOR` set (on + /// macOS/aarch64 with the feature) runs the corpus through the hypervisor at 4K; + /// otherwise the generic sandbox at the native page size. Diagnostic-only — the + /// env var is never set by the normal test runs. + pub(super) fn select_sandbox() -> (crate::SandboxKind, u32) { + #[cfg(all(target_os = "macos", target_arch = "aarch64", feature = "hypervisor-sandbox"))] + { + if std::env::var("POLKAVM_TEST_HYPERVISOR").is_ok() { + return (crate::SandboxKind::Hypervisor, 4096); + } + } + (crate::SandboxKind::Generic, get_native_page_size().try_into().unwrap()) + } + + /// Assembles `code` (one exported entry block, terminated by `trap` or `ret`), runs it on the + /// compiler + selected sandbox with `inputs` preloaded, and returns the interrupt and instance. + fn run(isa: InstructionSetKind, rw_size: u32, code: &[Instruction], inputs: &[(Reg, RegValue)]) -> (InterruptKind, RawInstance) { + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(isa); + if rw_size > 0 { + builder.set_rw_data_size(rw_size); + } + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(code, &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let (sandbox, page_size) = select_sandbox(); + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(sandbox)); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(page_size); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + for &(reg, value) in inputs { + instance.set_reg(reg, value); + } + instance.set_next_program_counter(entry); + (instance.run().unwrap(), instance) + } + + fn run32(code: &[Instruction], inputs: &[(Reg, RegValue)]) -> RawInstance { + let (interrupt, instance) = run(InstructionSetKind::Latest32, 0, code, inputs); + assert!(matches!(interrupt, InterruptKind::Trap), "unexpected interrupt: {interrupt:?}"); + instance + } + + #[test] + fn far_jump_table_is_reachable() { + // The jump table sits past the end of the code, so with >1 MiB of code the indirect-jump + // `adrp` needs a multi-page offset; `adr` (±1 MiB) used to panic here. Block 0 is a `ret`, + // which goes through the table, so this also checks the computed base is right. + let _ = env_logger::try_init(); + let mut code = vec![asm::ret()]; + for _ in 0..200_000 { + code.push(asm::add_imm_32(A0, A0, 1)); + } + code.push(asm::trap()); + + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest64); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(&code, &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let (sandbox, page_size) = select_sandbox(); + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(sandbox)); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(page_size); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let native_len = module.machine_code().map_or(0, |code| code.len()); + assert!(native_len > 0x100000, "test is pointless below 1 MiB of code: {native_len}"); + + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_next_program_counter(entry); + assert!(matches!(instance.run().unwrap(), InterruptKind::Finished)); + } + + #[test] + fn add_trap() { + // Validates entry/exit, sysenter register restore, the add handler and the trap trampoline. + let i = run32(&[asm::add_32(A0, A0, A1), asm::trap()], &[(A0, 1), (A1, 10)]); + assert_eq!(i.reg(A0), 11); + } + + #[test] + fn arithmetic() { + let i = run32( + &[ + asm::add_32(S0, A0, A1), + asm::sub_32(S1, A0, A1), + asm::mul_32(A2, A0, A1), + asm::add_imm_32(T0, A0, 5), + asm::negate_and_add_imm_32(T1, A0, 100), + asm::trap(), + ], + &[(A0, 20), (A1, 6)], + ); + assert_eq!(i.reg(S0), 26); + assert_eq!(i.reg(S1), 14); + assert_eq!(i.reg(A2), 120); + assert_eq!(i.reg(T0), 25); + assert_eq!(i.reg(T1), 80); // 100 - 20 + } + + #[test] + fn division() { + let i = run32( + &[asm::div_unsigned_32(S0, A0, A1), asm::rem_unsigned_32(S1, A0, A1), asm::trap()], + &[(A0, 20), (A1, 6)], + ); + assert_eq!(i.reg(S0), 3); + assert_eq!(i.reg(S1), 2); + drop(i); // release the sandbox before the next run (one hypervisor VM per process) + + // Divide-by-zero: RISC-V yields an all-ones quotient and a remainder equal to the dividend. + let i = run32( + &[asm::div_unsigned_32(S0, A0, A1), asm::rem_unsigned_32(S1, A0, A1), asm::trap()], + &[(A0, 7), (A1, 0)], + ); + assert_eq!(i.reg(S0), 0xffff_ffff); + assert_eq!(i.reg(S1), 7); + drop(i); + + // Signed overflow (INT_MIN / -1): quotient INT_MIN, remainder 0. + let i = run32( + &[asm::div_signed_32(S0, A0, A1), asm::rem_signed_32(S1, A0, A1), asm::trap()], + &[(A0, 0x8000_0000), (A1, 0xffff_ffff)], + ); + assert_eq!(i.reg(S0), 0x8000_0000); + assert_eq!(i.reg(S1), 0); + } + + #[test] + fn logical() { + let i = run32( + &[ + asm::and(S0, A0, A1), + asm::or(S1, A0, A1), + asm::xor(A2, A0, A1), + asm::and_inverted(T0, A0, A1), + asm::xnor(T1, A0, A1), + asm::trap(), + ], + &[(A0, 0b1100), (A1, 0b1010)], + ); + assert_eq!(i.reg(S0), 0b1000); + assert_eq!(i.reg(S1), 0b1110); + assert_eq!(i.reg(A2), 0b0110); + assert_eq!(i.reg(T0), 0b0100); // a0 & ~a1 + assert_eq!(i.reg(T1), 0xffff_fff9); // ~(a0 ^ a1), 32-bit + } + + #[test] + fn shifts() { + let i = run32( + &[ + asm::shift_logical_left_imm_32(S0, A0, 4), + asm::shift_logical_right_imm_32(S1, A0, 4), + asm::shift_arithmetic_right_imm_32(A3, A1, 4), + asm::shift_logical_left_32(T0, A0, A2), + asm::trap(), + ], + &[(A0, 0xf0), (A1, 0x8000_0000), (A2, 8)], + ); + assert_eq!(i.reg(S0), 0xf00); + assert_eq!(i.reg(S1), 0xf); + assert_eq!(i.reg(A3), 0xf800_0000); + assert_eq!(i.reg(T0), 0xf000); + } + + #[test] + fn compare_and_select() { + let i = run32( + &[ + asm::set_less_than_unsigned(S0, A0, A1), + asm::set_less_than_signed(S1, A1, A0), + asm::minimum(A2, A0, A1), + asm::maximum_unsigned(T0, A0, A1), + asm::set_greater_than_unsigned_imm(T1, A1, 5), + asm::trap(), + ], + &[(A0, 5), (A1, 10)], + ); + assert_eq!(i.reg(S0), 1); + assert_eq!(i.reg(S1), 0); + assert_eq!(i.reg(A2), 5); + assert_eq!(i.reg(T0), 10); + assert_eq!(i.reg(T1), 1); + drop(i); // release the sandbox before the next run (one hypervisor VM per process) + + // `cmov` keeps the destination's previous value when the condition does not hold. + let i = run32( + &[ + asm::load_imm(S0, 99), + asm::cmov_if_zero(S0, A0, A2), + asm::load_imm(S1, 99), + asm::cmov_if_not_zero(S1, A0, A2), + asm::trap(), + ], + &[(A0, 5), (A2, 0)], + ); + assert_eq!(i.reg(S0), 5); + assert_eq!(i.reg(S1), 99); + } + + #[test] + fn bit_ops() { + let i = run32( + &[ + asm::count_leading_zero_bits_32(S0, A0), + asm::count_trailing_zero_bits_32(S1, A1), + asm::sign_extend_8(A5, A2), + asm::zero_extend_16(T0, A3), + asm::reverse_byte(T1, A4), + asm::trap(), + ], + &[(A0, 1), (A1, 0x100), (A2, 0x80), (A3, 0xffff_1234), (A4, 0x1234_5678)], + ); + assert_eq!(i.reg(S0), 31); + assert_eq!(i.reg(S1), 8); + assert_eq!(i.reg(A5), 0xffff_ff80); + assert_eq!(i.reg(T0), 0x1234); + assert_eq!(i.reg(T1), 0x7856_3412); + } + + #[test] + fn memory_roundtrip() { + let _ = env_logger::try_init(); + let (sandbox, page) = select_sandbox(); + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(sandbox)); + // Creating the engine initializes the native page size queried below. + let engine = Engine::new(&config).unwrap(); + let addr = i32::try_from(MemoryMapBuilder::new(page).rw_data_size(0x4000).build().unwrap().rw_data_address()).unwrap(); + + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest64); + builder.set_rw_data_size(0x4000); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code( + &[ + asm::store_imm_u32(addr, 0x1234_5678), + asm::load_u32(S0, addr), + asm::load_u8(S1, addr), + asm::load_u16(A2, addr), + asm::load_imm(T2, addr), + asm::load_indirect_u32(T0, T2, 0), + asm::store_u8(A0, addr + 8), + asm::load_u8(T1, addr + 8), + asm::trap(), + ], + &[], + ); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(page); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut i = module.instantiate().unwrap(); + i.set_reg(Reg::RA, crate::RETURN_TO_HOST); + i.set_reg(A0, 0xab); + i.set_next_program_counter(entry); + assert!(matches!(i.run().unwrap(), InterruptKind::Trap)); + assert_eq!(i.reg(S0), 0x1234_5678); + assert_eq!(i.reg(S1), 0x78); + assert_eq!(i.reg(A2), 0x5678); + assert_eq!(i.reg(T0), 0x1234_5678); + assert_eq!(i.reg(T1), 0xab); + } + + #[test] + fn conditional_branch() { + // block 0: branch; block 1 (fallthrough): S0 = 100; block 2 (taken): S0 = 200. + let code = &[ + asm::branch_less_unsigned(A0, A1, 2), + asm::load_imm(S0, 100), + asm::trap(), + asm::load_imm(S0, 200), + asm::trap(), + ]; + assert_eq!(run32(code, &[(A0, 5), (A1, 10)]).reg(S0), 200); // taken + assert_eq!(run32(code, &[(A0, 10), (A1, 5)]).reg(S0), 100); // not taken + } + + #[test] + fn fallthrough_trap_gas_64() { + // Mirrors the differential fuzzer: interpreter + recompiler instances of `[fallthrough, trap]` + // (Latest64 + Sync gas), engines/modules dropped before running. + let blob = { + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest64); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(&[asm::fallthrough(), asm::trap()], &[]); + ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap() + }; + + let make = |backend, generic: bool| { + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(backend)); + let mut page = get_native_page_size().try_into().unwrap(); + if generic { + let (sandbox, sandbox_page) = select_sandbox(); + config.set_sandbox(Some(sandbox)); + page = sandbox_page; + } + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(page); + module_config.set_gas_metering(Some(GasMeteringKind::Sync)); + let module = Module::from_blob(&engine, &module_config, blob.clone()).unwrap(); + let mut i = module.instantiate().unwrap(); + i.set_gas(10000); + i.set_next_program_counter(ProgramCounter(0)); + i.set_reg(Reg::RA, crate::RETURN_TO_HOST); + i + }; + + let mut interp = make(BackendKind::Interpreter, false); + let mut recomp = make(BackendKind::Compiler, true); + assert!(matches!(interp.run().unwrap(), InterruptKind::Trap)); + assert!(matches!(recomp.run().unwrap(), InterruptKind::Trap)); + } + + #[test] + fn return_to_host() { + // `ret` exits through the jump table back to the host (InterruptKind::Finished). + let (interrupt, i) = run( + InstructionSetKind::Latest32, + 0, + &[asm::add_32(A0, A0, A1), asm::ret()], + &[(A0, 1), (A1, 10)], + ); + assert!(matches!(interrupt, InterruptKind::Finished), "unexpected interrupt: {interrupt:?}"); + assert_eq!(i.reg(A0), 11); + } + + #[test] + fn arithmetic_64bit() { + let (interrupt, i) = run( + InstructionSetKind::Latest64, + 0, + &[ + asm::load_imm64(S0, 0x1_0000_0000), + asm::add_64(S1, S0, S0), + asm::mul_64(A2, A0, A1), + asm::trap(), + ], + &[(A0, 0x1_0000_0000), (A1, 3)], + ); + assert!(matches!(interrupt, InterruptKind::Trap), "unexpected interrupt: {interrupt:?}"); + assert_eq!(i.reg(S0), 0x1_0000_0000); + assert_eq!(i.reg(S1), 0x2_0000_0000); + assert_eq!(i.reg(A2), 0x3_0000_0000); + } + + #[test] + fn gas_metering() { + let (sandbox, page) = select_sandbox(); + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(sandbox)); + let engine = Engine::new(&config).unwrap(); + + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(&[asm::add_32(A0, A0, A1), asm::trap()], &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(page); + module_config.set_gas_metering(Some(GasMeteringKind::Sync)); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + // Enough gas: the block runs and gas is charged. + let mut instance = module.instantiate().unwrap(); + instance.set_gas(1000); + instance.prepare_call_typed(entry, (1u32, 10u32)); + assert!(matches!(instance.run().unwrap(), InterruptKind::Trap)); + assert_eq!(instance.reg(A0), 11); + assert!(instance.gas() < 1000 && instance.gas() >= 0, "gas not charged: {}", instance.gas()); + drop(instance); // release the sandbox before the next run (one hypervisor VM per process) + + // No gas: the metering stub underflows and traps into NotEnoughGas via the UDF→SIGILL path. + let mut instance = module.instantiate().unwrap(); + instance.set_gas(0); + instance.prepare_call_typed(entry, (1u32, 10u32)); + assert!(matches!(instance.run().unwrap(), InterruptKind::NotEnoughGas)); + } +} + fn fallback_hostcall_handler_works(config: Config, isa: InstructionSetKind) { let _ = env_logger::try_init(); let blob = basic_test_blob(isa); @@ -929,6 +1380,53 @@ fn out_of_range_execution(engine_config: Config, isa: InstructionSetKind) { assert_eq!(instance.program_counter(), Some(offsets[2])); } +/// Known gas divergence at `pc == code_len` for non-terminating last blocks. +#[cfg(target_os = "linux")] +#[test] +#[ignore = "known interpreter/recompiler gas divergence at implicit fall-off-end PC"] +fn out_of_range_pc_charges_consistent_gas_across_backends() { + fn run(backend: BackendKind) -> (InterruptKind, i64) { + let mut engine_config = Config::default(); + engine_config.set_backend(Some(backend)); + if backend == BackendKind::Compiler { + engine_config.set_sandbox(Some(crate::SandboxKind::Linux)); + engine_config.set_worker_count(1); + } + let engine = Engine::new(&engine_config).unwrap(); + + let mut module_config = ModuleConfig::new(); + module_config.set_gas_metering(Some(GasMeteringKind::Sync)); + + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(&[asm::fallthrough()], &[]); + + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let mut instance = module.instantiate().unwrap(); + + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_gas(10_000); + instance.set_next_program_counter(ProgramCounter(1)); // past the only block + + let interrupt = instance.run().unwrap(); + (interrupt, instance.gas()) + } + + let (interp_int, interp_gas) = run(BackendKind::Interpreter); + let (comp_int, comp_gas) = run(BackendKind::Compiler); + + // Both backends agree the run trapped. + assert_eq!(interp_int, comp_int, "interrupt kind diverged"); + assert_eq!(interp_int, InterruptKind::Trap); + + // ...but they currently disagree on the gas cost. + assert_eq!( + interp_gas, comp_gas, + "gas accounting diverged for an out-of-range PC: interpreter left {interp_gas}, recompiler left {comp_gas}" + ); +} + fn jump_into_middle_of_basic_block_from_outside(engine_config: Config, isa: InstructionSetKind) { let _ = env_logger::try_init(); let engine = Engine::new(&engine_config).unwrap(); @@ -1480,6 +1978,9 @@ fn jump_to_an_injected_invalid_instruction(engine_config: Config, isa: Instructi } fn jump_indirect_simple(engine_config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&engine_config) { + return; + } let _ = env_logger::try_init(); let engine = Engine::new(&engine_config).unwrap(); let mut builder = ProgramBlobBuilder::new(isa); @@ -1814,6 +2315,9 @@ fn dynamic_paging_stress_test(_engine_config: Config, _: InstructionSetKind) {} #[cfg(feature = "std")] fn dynamic_paging_stress_test(mut engine_config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&engine_config) { + return; + } let _lock = StressTestLock::new(); let _ = env_logger::try_init(); engine_config.set_allow_dynamic_paging(true); @@ -2016,7 +2520,7 @@ fn dynamic_paging_read_at_top_of_address_space(mut engine_config: Config, isa: I instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); instance.set_next_program_counter(offsets[0]); let segfault = expect_segfault(instance.run().unwrap()); - assert_eq!(segfault.page_address, 0xfffff000); + assert_eq!(segfault.page_address, !(page_size - 1)); // 0xfffff000 at 4K } fn dynamic_paging_read_with_upper_bits_set(mut engine_config: Config, isa: InstructionSetKind) { @@ -2152,9 +2656,12 @@ fn dynamic_paging_write_at_page_boundary_with_no_pages(mut engine_config: Config let engine = Engine::new(&engine_config).unwrap(); let page_size = get_native_page_size() as u32; + // Straddle the boundary between the first and second page (page-size-relative; == 0x10ffe/0x11000 at 4K). + let second_page = 0x10000 + page_size; + let straddle = second_page - 2; let mut builder = ProgramBlobBuilder::new(isa); builder.add_export_by_basic_block(0, b"main"); - builder.set_code(&[asm::store_imm_u32(0x10ffe, 0x12345678), asm::ret()], &[]); + builder.set_code(&[asm::store_imm_u32(cast(straddle).to_i32_or_panic(), 0x12345678), asm::ret()], &[]); let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); let mut module_config = ModuleConfig::new(); @@ -2173,14 +2680,14 @@ fn dynamic_paging_write_at_page_boundary_with_no_pages(mut engine_config: Config .unwrap(); let segfault = expect_segfault(instance.run().unwrap()); - assert_eq!(segfault.page_address, 0x11000); - assert_eq!(instance.read_memory(0x10ffe, 2).unwrap(), vec![0, 0]); + assert_eq!(segfault.page_address, second_page); + assert_eq!(instance.read_memory(straddle, 2).unwrap(), vec![0, 0]); instance - .zero_memory_with_memory_protection(0x11000, page_size, MemoryProtection::ReadWrite) + .zero_memory_with_memory_protection(second_page, page_size, MemoryProtection::ReadWrite) .unwrap(); match_interrupt!(instance.run().unwrap(), InterruptKind::Finished); - assert_eq!(instance.read_memory(0x10ffe, 2).unwrap(), vec![0x78, 0x56]); + assert_eq!(instance.read_memory(straddle, 2).unwrap(), vec![0x78, 0x56]); } fn dynamic_paging_write_at_page_boundary_with_first_page(mut engine_config: Config, isa: InstructionSetKind) { @@ -2190,9 +2697,11 @@ fn dynamic_paging_write_at_page_boundary_with_first_page(mut engine_config: Conf let engine = Engine::new(&engine_config).unwrap(); let page_size = get_native_page_size() as u32; + let second_page = 0x10000 + page_size; + let straddle = second_page - 2; let mut builder = ProgramBlobBuilder::new(isa); builder.add_export_by_basic_block(0, b"main"); - builder.set_code(&[asm::store_imm_u32(0x10ffe, 0x12345678), asm::ret()], &[]); + builder.set_code(&[asm::store_imm_u32(cast(straddle).to_i32_or_panic(), 0x12345678), asm::ret()], &[]); let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); let mut module_config = ModuleConfig::new(); @@ -2209,14 +2718,14 @@ fn dynamic_paging_write_at_page_boundary_with_first_page(mut engine_config: Conf .unwrap(); let segfault = expect_segfault(instance.run().unwrap()); - assert_eq!(segfault.page_address, 0x11000); - assert_eq!(instance.read_memory(0x10ffe, 2).unwrap(), vec![0, 0]); + assert_eq!(segfault.page_address, second_page); + assert_eq!(instance.read_memory(straddle, 2).unwrap(), vec![0, 0]); instance - .zero_memory_with_memory_protection(0x11000, page_size, MemoryProtection::ReadWrite) + .zero_memory_with_memory_protection(second_page, page_size, MemoryProtection::ReadWrite) .unwrap(); match_interrupt!(instance.run().unwrap(), InterruptKind::Finished); - assert_eq!(instance.read_memory(0x10ffe, 2).unwrap(), vec![0x78, 0x56]); + assert_eq!(instance.read_memory(straddle, 2).unwrap(), vec![0x78, 0x56]); } fn dynamic_paging_write_at_page_boundary_with_second_page(mut engine_config: Config, isa: InstructionSetKind) { @@ -2226,9 +2735,11 @@ fn dynamic_paging_write_at_page_boundary_with_second_page(mut engine_config: Con let engine = Engine::new(&engine_config).unwrap(); let page_size = get_native_page_size() as u32; + let second_page = 0x10000 + page_size; + let straddle = second_page - 2; let mut builder = ProgramBlobBuilder::new(isa); builder.add_export_by_basic_block(0, b"main"); - builder.set_code(&[asm::store_imm_u32(0x10ffe, 0x12345678), asm::ret()], &[]); + builder.set_code(&[asm::store_imm_u32(cast(straddle).to_i32_or_panic(), 0x12345678), asm::ret()], &[]); let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); let mut module_config = ModuleConfig::new(); @@ -2241,18 +2752,18 @@ fn dynamic_paging_write_at_page_boundary_with_second_page(mut engine_config: Con instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); instance.set_next_program_counter(offsets[0]); instance - .zero_memory_with_memory_protection(0x11000, page_size, MemoryProtection::ReadWrite) + .zero_memory_with_memory_protection(second_page, page_size, MemoryProtection::ReadWrite) .unwrap(); let segfault = expect_segfault(instance.run().unwrap()); assert_eq!(segfault.page_address, 0x10000); - assert_eq!(instance.read_memory(0x11000, 2).unwrap(), vec![0, 0]); + assert_eq!(instance.read_memory(second_page, 2).unwrap(), vec![0, 0]); instance .zero_memory_with_memory_protection(0x10000, page_size, MemoryProtection::ReadWrite) .unwrap(); match_interrupt!(instance.run().unwrap(), InterruptKind::Finished); - assert_eq!(instance.read_memory(0x11000, 2).unwrap(), vec![0x34, 0x12]); + assert_eq!(instance.read_memory(second_page, 2).unwrap(), vec![0x34, 0x12]); } fn dynamic_paging_change_written_value_and_address_during_segfault(mut engine_config: Config, isa: InstructionSetKind) { @@ -2310,19 +2821,21 @@ fn dynamic_paging_cancel_segfault_by_changing_address(mut engine_config: Config, let module = Module::from_blob(&engine, &module_config, blob).unwrap(); let offsets: Vec<_> = module.blob().instructions().map(|inst| inst.offset).collect(); + // Page-size agnostic: 0x11000 at 4K. + let other_page = 0x10000 + page_size; let mut instance = module.instantiate().unwrap(); instance - .zero_memory_with_memory_protection(0x11000, page_size, MemoryProtection::ReadWrite) + .zero_memory_with_memory_protection(other_page, page_size, MemoryProtection::ReadWrite) .unwrap(); instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); instance.set_next_program_counter(offsets[0]); instance.set_reg(Reg::A0, 0x10000); let segfault = expect_segfault(instance.run().unwrap()); assert_eq!(segfault.page_address, 0x10000); - instance.set_reg(Reg::A0, 0x11000); + instance.set_reg(Reg::A0, u64::from(other_page)); match_interrupt!(instance.run().unwrap(), InterruptKind::Finished); assert_eq!( - instance.read_memory(0x11000 + store_offset as u32, 4).unwrap(), + instance.read_memory(other_page + store_offset as u32, 4).unwrap(), vec![0x78, 0x56, 0x34, 0x12] ); } @@ -2336,6 +2849,7 @@ fn dynamic_paging_worker_recycle_turn_dynamic_paging_on_and_off(mut engine_confi let engine = Engine::new(&engine_config).unwrap(); let page_size = get_native_page_size() as u32; + let next_page = 0x20000 + page_size; // 0x21000 at 4K; a separate page from the 0x20000 target let mut builder = ProgramBlobBuilder::new(isa); builder.add_export_by_basic_block(0, b"main"); builder.set_rw_data_size(1); @@ -2370,17 +2884,17 @@ fn dynamic_paging_worker_recycle_turn_dynamic_paging_on_and_off(mut engine_confi assert_eq!(segfault.page_address, 0x20000); assert_eq!(segfault.page_size, page_size); let segfault = expect_segfault(instance.run().unwrap()); - assert_out_of_range_access(instance.read_u32(0x21000), 0x21000, 4); + assert_out_of_range_access(instance.read_u32(next_page), next_page, 4); instance .zero_memory_with_memory_protection(segfault.page_address, page_size + 4, MemoryProtection::ReadWrite) .unwrap(); match_interrupt!(instance.run().unwrap(), InterruptKind::Finished); assert_eq!(instance.read_u32(0x20000).unwrap(), 0x12345678); - assert_eq!(instance.read_u32(0x21000).unwrap(), 0); + assert_eq!(instance.read_u32(next_page).unwrap(), 0); instance.set_next_program_counter(ProgramCounter(0)); match_interrupt!(instance.run().unwrap(), InterruptKind::Finished); } else { - assert_out_of_range_access(instance.read_u32(0x21000), 0x21000, 4); + assert_out_of_range_access(instance.read_u32(next_page), next_page, 4); assert_eq!(instance.read_u32(0x20000).unwrap(), 0); match_interrupt!(instance.run().unwrap(), InterruptKind::Finished); assert_eq!(instance.read_u32(0x20000).unwrap(), 0x12345678); @@ -2448,13 +2962,14 @@ fn dynamic_paging_change_program_counter_during_segfault(mut engine_config: Conf let engine = Engine::new(&engine_config).unwrap(); let page_size = get_native_page_size() as u32; + let second_page = 0x10000 + page_size; // 0x11000 at 4K; a separate page let mut builder = ProgramBlobBuilder::new(isa); builder.add_export_by_basic_block(0, b"main"); builder.set_code( &[ asm::store_imm_u32(0x10000, 1), asm::ret(), - asm::store_imm_u32(0x11000, 2), + asm::store_imm_u32(cast(second_page).to_i32_or_panic(), 2), asm::ret(), ], &[], @@ -2475,12 +2990,12 @@ fn dynamic_paging_change_program_counter_during_segfault(mut engine_config: Conf instance.set_next_program_counter(offsets[2]); let segfault = expect_segfault(instance.run().unwrap()); - assert_eq!(segfault.page_address, 0x11000); + assert_eq!(segfault.page_address, second_page); instance .zero_memory_with_memory_protection(segfault.page_address, page_size, MemoryProtection::ReadWrite) .unwrap(); match_interrupt!(instance.run().unwrap(), InterruptKind::Finished); - assert_eq!(instance.read_u32(0x11000).unwrap(), 2); + assert_eq!(instance.read_u32(second_page).unwrap(), 2); } fn dynamic_paging_run_out_of_gas(mut engine_config: Config, isa: InstructionSetKind) { @@ -2519,6 +3034,9 @@ fn dynamic_paging_receive_from_another_thread_and_run(_: Config, _: InstructionS #[cfg(feature = "std")] fn dynamic_paging_receive_from_another_thread_and_run(mut engine_config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&engine_config) { + return; + } engine_config.set_allow_dynamic_paging(true); let _ = env_logger::try_init(); @@ -2565,6 +3083,9 @@ fn dynamic_paging_instantiate_on_another_thread(_: Config, _: InstructionSetKind #[cfg(feature = "std")] fn dynamic_paging_instantiate_on_another_thread(mut engine_config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&engine_config) { + return; + } engine_config.set_allow_dynamic_paging(true); let _ = env_logger::try_init(); @@ -2633,6 +3154,9 @@ fn dynamic_paging_parallel_page_fault_stress_test(_: Config, _: InstructionSetKi #[cfg(feature = "std")] fn dynamic_paging_parallel_page_fault_stress_test(mut engine_config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&engine_config) { + return; + } let _lock = StressTestLock::new(); engine_config.set_allow_dynamic_paging(true); @@ -2987,7 +3511,24 @@ fn pinky_standard(config: Config, isa: InstructionSetKind) { pinky_impl(config, isa) } +// The generic sandbox enforces guest page protections with `mprotect`, so its granularity is the +// *host* page size. On macOS (16K) a 4K-page module is under-protected by up to 12K: accesses past a +// guest page boundary don't fault until the next 16K boundary, so tests asserting an out-of-bounds +// trap fail. They pass on 4K-page hosts (including the production target); running them faithfully +// on macOS needs the hypervisor sandbox, whose stage-1 MMU protects at a real 4K granule. +fn skip_on_16k_native_page(config: &Config) -> bool { + // `get_native_page_size()` returns 4096 when the compiler/sandbox isn't built, so this is false there. + if_compiler_is_supported! { + { crate::sandbox::init_native_page_size(); } else {} + } + cfg!(target_os = "macos") && config.backend() == Some(BackendKind::Compiler) && get_native_page_size() > 0x1000 +} + fn pinky_impl(config: Config, isa: InstructionSetKind) { + if skip_on_16k_native_page(&config) { + return; + } + if (config.backend() == Some(crate::BackendKind::Interpreter) && cfg!(debug_assertions)) || config.crosscheck() { return; // Too slow. } @@ -2999,6 +3540,8 @@ fn pinky_impl(config: Config, isa: InstructionSetKind) { let mut module_config = ModuleConfig::default(); if config.allow_dynamic_paging() { module_config.set_dynamic_paging(true); + // Dynamic paging requires the module page size to match the native one (== 4096 on x86). + module_config.set_page_size(get_native_page_size() as u32); } let module = Module::from_blob(&engine, &module_config, blob).unwrap(); let linker: Linker = Linker::new(); @@ -3178,6 +3721,9 @@ fn invalid_instruction_after_fallthrough(engine_config: Config, isa: Instruction } fn invalid_branch_target(engine_config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&engine_config) { + return; + } let _ = env_logger::try_init(); let engine = Engine::new(&engine_config).unwrap(); let mut builder = ProgramBlobBuilder::new(isa); @@ -3549,6 +4095,9 @@ fn aux_data_accessible_area(config: Config, isa: InstructionSetKind) { } fn access_memory_from_host(config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&config) { + return; + } let _ = env_logger::try_init(); let engine = Engine::new(&config).unwrap(); let page_size = get_native_page_size() as u32; @@ -3674,6 +4223,9 @@ fn access_memory_from_host(config: Config, isa: InstructionSetKind) { } fn access_memory_from_within(config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&config) { + return; + } let _ = env_logger::try_init(); let engine = Engine::new(&config).unwrap(); @@ -3990,6 +4542,9 @@ fn interpreter_guest_memory_limit() { } fn write_read_memory_from_host(config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&config) { + return; + } let _ = env_logger::try_init(); let engine = Engine::new(&config).unwrap(); @@ -4366,6 +4921,9 @@ fn test_blob_out_of_bounds_memory_access_generates_a_trap(args: TestBlobArgs) { } fn test_blob_call_sbrk_impl(args: TestBlobArgs, mut call_sbrk: impl FnMut(&mut TestInstance, u32) -> u32) { + if skip_on_16k_native_page(&args.config) { + return; + } let elf = args.get_test_program(); let mut i = TestInstance::new(&args, elf); let memory_map = i.module.memory_map().clone(); @@ -4614,6 +5172,9 @@ fn test_blob_max_zero_const_64(args: TestBlobArgs) { } fn test_asm_reloc_add_sub(config: Config, isa: InstructionSetKind, optimize: bool) { + if skip_on_16k_native_page(&config) { + return; + } let args = TestBlobArgs { config, isa, @@ -4637,6 +5198,9 @@ fn test_asm_reloc_add_sub(config: Config, isa: InstructionSetKind, optimize: boo } fn test_asm_reloc_hi_lo(config: Config, isa: InstructionSetKind, optimize: bool) { + if skip_on_16k_native_page(&config) { + return; + } let args = TestBlobArgs { config, isa, @@ -5109,6 +5673,9 @@ fn memset_basic(config: Config, isa: InstructionSetKind) { if !isa.supports_opcode(Opcode::memset) { return; } + if skip_on_16k_native_page(&config) { + return; + } let _ = env_logger::try_init(); @@ -5238,6 +5805,9 @@ fn memset_basic(config: Config, isa: InstructionSetKind) { } fn memset_with_dynamic_paging(mut config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&config) { + return; + } if !isa.supports_opcode(Opcode::memset) { return; } @@ -5856,6 +6426,9 @@ fn spawn_stress_test(_config: Config, _: InstructionSetKind) {} #[cfg(feature = "std")] fn spawn_stress_test(mut config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&config) { + return; + } let _lock = StressTestLock::new(); let _ = env_logger::try_init(); @@ -5905,6 +6478,9 @@ fn spawn_stress_test(mut config: Config, isa: InstructionSetKind) { } fn spawn_inner_vm(config: Config, isa: InstructionSetKind) { + if skip_if_single_instance_sandbox(&config) { + return; + } let _ = env_logger::try_init(); let mut builder = ProgramBlobBuilder::new(isa); @@ -6303,3 +6879,509 @@ assert_send_sync! { crate::ModuleConfig, crate::ProgramBlob, } + +// Smoke test for the Hypervisor sandbox execution path. Requires the test process to be +// codesigned with com.apple.security.hypervisor (see ci/hypervisor/sign-and-run.sh); without +// it `hv_vm_create` returns HV_DENIED and instantiation fails. +#[cfg(all(target_os = "macos", target_arch = "aarch64", feature = "hypervisor-sandbox"))] +mod aarch64_hypervisor { + use super::*; + use crate::{RawInstance, RegValue}; + use polkavm_common::program::Instruction; + + /// `hv_vm_create` needs the `com.apple.security.hypervisor` entitlement, which a plain + /// `cargo test` binary doesn't have. Without it there is nothing to exercise, so skip instead of + /// failing; `ci/jobs/build-and-test-macos-hypervisor.sh` runs these against a signed binary. + fn hypervisor_unavailable() -> bool { + static AVAILABLE: std::sync::OnceLock = std::sync::OnceLock::new(); + let available = *AVAILABLE.get_or_init(|| crate::sandbox::hypervisor::Vm::new().is_ok()); + if !available { + eprintln!("skipping: the test binary lacks the hypervisor entitlement"); + } + !available + } + + fn run_hv(code: &[Instruction], inputs: &[(Reg, RegValue)]) -> (InterruptKind, RawInstance) { + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(code, &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); // the hypervisor guest runs at 4K + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + for &(reg, value) in inputs { + instance.set_reg(reg, value); + } + instance.set_next_program_counter(entry); + (instance.run().unwrap(), instance) + } + + /// `reset_memory` must restore the initial contents *and* leave the instance runnable: the + /// static map protections have to survive the reset. + #[test] + fn hv_reset_memory_keeps_instance_usable() { + if hypervisor_unavailable() { + return; + } + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.set_rw_data_size(4096); + builder.set_code(&[asm::store_imm_indirect_u32(A0, 0, 0x1234_5678), asm::ret()], &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + let rw = module.memory_map().rw_data_address(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_reg(A0, u64::from(rw)); + instance.set_next_program_counter(entry); + assert!(matches!(instance.run().unwrap(), InterruptKind::Finished)); + assert_eq!(instance.read_memory(rw, 4).unwrap(), 0x1234_5678u32.to_le_bytes()); + + instance.reset_memory().unwrap(); + assert_eq!(instance.read_memory(rw, 4).unwrap(), [0, 0, 0, 0]); + + // The guest must still be able to write there afterwards. + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_reg(A0, u64::from(rw)); + instance.set_next_program_counter(entry); + let interrupt = instance.run().unwrap(); + assert!(matches!(interrupt, InterruptKind::Finished), "after reset: {interrupt:?}"); + assert_eq!(instance.read_memory(rw, 4).unwrap(), 0x1234_5678u32.to_le_bytes()); + } + + /// `set_accessible_aux_size` must actually bind the guest: reads inside the accessible prefix + /// work, reads past it trap, and writes are refused even inside it (aux data is read-only). + #[test] + fn hv_aux_data_is_bounded() { + if hypervisor_unavailable() { + return; + } + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(&[asm::load_indirect_u8(S0, A0, 0), asm::ret()], &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); + module_config.set_aux_data_size(4096 * 2); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + let aux = module.memory_map().aux_data_address(); + + let mut instance = module.instantiate().unwrap(); + // Only the first page is accessible to the guest. + instance.set_accessible_aux_size(4096).unwrap(); + instance.write_memory(aux, &[0x5a]).unwrap(); // host can always write it + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + + // Inside the accessible prefix: the load succeeds. + instance.set_reg(A0, u64::from(aux)); + instance.set_next_program_counter(entry); + let interrupt = instance.run().unwrap(); + assert!(matches!(interrupt, InterruptKind::Finished), "in-bounds aux read: {interrupt:?}"); + assert_eq!(instance.reg(S0), 0x5a); + + // Past it: the load traps. + instance.set_reg(A0, u64::from(aux + 4096)); + instance.set_next_program_counter(entry); + let interrupt = instance.run().unwrap(); + assert!(matches!(interrupt, InterruptKind::Trap), "out-of-bounds aux read: {interrupt:?}"); + + // Shrinking to zero closes the prefix too. + instance.set_accessible_aux_size(0).unwrap(); + instance.set_reg(A0, u64::from(aux)); + instance.set_next_program_counter(entry); + let interrupt = instance.run().unwrap(); + assert!( + matches!(interrupt, InterruptKind::Trap), + "aux read after shrinking to 0: {interrupt:?}" + ); + + assert!(instance.set_accessible_aux_size(4096 * 3).is_err()); // beyond the region + } + + /// Revoking a page the guest has already touched must actually stop it: the stage-1 TLB holds + /// the old entry, so the host has to invalidate it before re-entering. Without the TLBI the + /// second store silently succeeds. + #[test] + fn hv_revoked_page_faults_again() { + if hypervisor_unavailable() { + return; + } + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.add_import(b"hostfn"); + // store; hand control back to the host; store again. + builder.set_code( + &[ + asm::store_imm_indirect_u32(A0, 0, 0x1111_1111), + asm::ecalli(0), + asm::store_imm_indirect_u32(A0, 0, 0x2222_2222), + asm::ret(), + ], + &[], + ); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + config.set_allow_dynamic_paging(true); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); + module_config.set_dynamic_paging(true); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_reg(A0, 0x20000); + instance.set_next_program_counter(entry); + + // Page in on the first fault, then let the store land -- this is what populates the TLB. + assert!(matches!(instance.run().unwrap(), InterruptKind::Segfault(_))); + instance + .zero_memory_with_memory_protection(0x20000, 4096, MemoryProtection::ReadWrite) + .unwrap(); + assert!(matches!(instance.run().unwrap(), InterruptKind::Ecalli(0))); + assert_eq!(instance.read_memory(0x20000, 4).unwrap(), 0x1111_1111u32.to_le_bytes()); + + // Revoke the page; the second store must fault instead of overwriting it. + instance.free_pages(0x20000, 4096).unwrap(); + instance.set_next_program_counter(instance.next_program_counter().unwrap()); + let interrupt = instance.run().unwrap(); + assert!( + matches!(interrupt, InterruptKind::Segfault(ref s) if s.page_address == 0x20000), + "revoked page did not fault: {interrupt:?}" + ); + assert!(instance.read_memory(0x20000, 4).is_err()); // unmapped again + } + + /// Sync metering runs out mid-guest: the `udf` in the metering stub must come back as + /// `NotEnoughGas` with the gas refunded and register state intact, exactly like the generic + /// sandbox (which the same program is checked against). + #[cfg(feature = "generic-sandbox")] // compares against the generic sandbox + #[test] + fn hv_gas_exhaustion_matches_generic() { + if hypervisor_unavailable() { + return; + } + let _ = env_logger::try_init(); + let code = [asm::add_imm_32(A0, A0, 1), asm::jump(0)]; // loops until the gas runs out + let mut results = Vec::new(); + for sandbox in [crate::SandboxKind::Generic, crate::SandboxKind::Hypervisor] { + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(&code, &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(sandbox)); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); + module_config.set_gas_metering(Some(crate::GasMeteringKind::Sync)); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_gas(100); + instance.set_next_program_counter(entry); + let interrupt = instance.run().unwrap(); + assert!(matches!(interrupt, InterruptKind::NotEnoughGas), "{sandbox:?}: {interrupt:?}"); + assert!(instance.gas() >= 0, "{sandbox:?}: gas left negative: {}", instance.gas()); + results.push((instance.gas(), instance.reg(A0), instance.program_counter())); + drop(instance); // one hypervisor VM per process + } + assert_eq!(results[0], results[1], "generic vs hypervisor: (gas, A0, pc) diverged"); + } + + #[test] + fn hv_add_trap() { + if hypervisor_unavailable() { + return; + } + // Smallest end-to-end path: entry stub -> add handler -> trap trampoline (hvc). + let (interrupt, i) = run_hv(&[asm::add_32(A0, A0, A1), asm::trap()], &[(A0, 1), (A1, 10)]); + assert!(matches!(interrupt, InterruptKind::Trap), "unexpected interrupt: {interrupt:?}"); + assert_eq!(i.reg(A0), 11); + } + + #[test] + fn hv_ret() { + if hypervisor_unavailable() { + return; + } + // Return-to-host: add handler -> ret (jump table[RETURN_TO_HOST]) -> sysreturn hvc. + let (interrupt, i) = run_hv(&[asm::add_32(A0, A0, A1), asm::ret()], &[(A0, 1), (A1, 10)]); + assert!(matches!(interrupt, InterruptKind::Finished), "unexpected interrupt: {interrupt:?}"); + assert_eq!(i.reg(A0), 11); + } + + #[test] + fn hv_ecall() { + if hypervisor_unavailable() { + return; + } + // Hostcall then resume: ecalli(0) -> Ecalli(0); resume -> add -> ret -> Finished. + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.add_import(b"hostfn"); + builder.set_code(&[asm::ecalli(0), asm::add_32(A0, A0, A1), asm::ret()], &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_reg(A0, 1); + instance.set_reg(A1, 10); + instance.set_next_program_counter(entry); + + let first = instance.run().unwrap(); + assert!(matches!(first, InterruptKind::Ecalli(0)), "unexpected interrupt: {first:?}"); + let second = instance.run().unwrap(); + assert!(matches!(second, InterruptKind::Finished), "unexpected interrupt: {second:?}"); + assert_eq!(instance.reg(A0), 11); + } + + #[test] + fn hv_sbrk() { + if hypervisor_unavailable() { + return; + } + // Guest sbrk crossing the heap threshold -> hvc #SBRK trampoline -> resume. + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(&[asm::sbrk(A0, A0), asm::ret()], &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let heap_base = module.memory_map().heap_base(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_reg(A0, 8192); // grow by two 4K pages (crosses the threshold) + instance.set_next_program_counter(entry); + + let interrupt = instance.run().unwrap(); + assert!(matches!(interrupt, InterruptKind::Finished), "unexpected interrupt: {interrupt:?}"); + // sbrk returns the new heap top. + assert_eq!(instance.reg(A0), u64::from(heap_base) + 8192); + + // The newly grown heap page is now accessible from the host. + instance.write_memory(heap_base, &[0xaa, 0xbb, 0xcc, 0xdd]).unwrap(); + assert_eq!(instance.read_memory(heap_base, 4).unwrap(), [0xaa, 0xbb, 0xcc, 0xdd]); + } + + #[test] + fn hv_segfault() { + if hypervisor_unavailable() { + return; + } + // Dynamic paging: load from an unmapped address -> guest EL1 abort -> hvc -> Segfault. + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + // Store to an unmapped guest address (base register A0 = 0x20000). + builder.set_code(&[asm::store_imm_indirect_u32(A0, 0, 0x1234), asm::ret()], &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + config.set_allow_dynamic_paging(true); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); + module_config.set_dynamic_paging(true); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_reg(A0, 0x20000); // unmapped guest address (>= null guard) + instance.set_next_program_counter(entry); + + let interrupt = instance.run().unwrap(); + match interrupt { + InterruptKind::Segfault(s) => { + assert_eq!(s.page_address, 0x20000); + assert_eq!(s.page_size, 4096); + } + other => panic!("unexpected interrupt: {other:?}"), + } + } + + #[test] + fn hv_dynamic_paging() { + if hypervisor_unavailable() { + return; + } + // Full round-trip: store to an unmapped page -> Segfault -> host maps + zeroes the + // page -> resume re-executes the store -> Finished -> value is readable. + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.set_code(&[asm::store_imm_indirect_u32(A0, 0, 0xAABB_CCDDu32 as i32), asm::ret()], &[]); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + config.set_allow_dynamic_paging(true); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); + module_config.set_dynamic_paging(true); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_reg(A0, 0x20000); + instance.set_next_program_counter(entry); + + // 1) The store faults on the unmapped page. + let s = match instance.run().unwrap() { + InterruptKind::Segfault(s) => s, + other => panic!("expected segfault, got {other:?}"), + }; + assert_eq!(s.page_address, 0x20000); + assert_eq!(s.page_size, 4096); + assert!(!s.is_write_protected); // page was absent, not read-only + assert!(!instance.is_memory_accessible(0x20000, 4, MemoryProtection::Read)); + assert!(instance.read_memory(0x20000, 4).is_err()); // unmapped -> error + + // 2) Map + zero the page. + instance + .zero_memory_with_memory_protection(0x20000, 4096, MemoryProtection::ReadWrite) + .unwrap(); + assert!(instance.is_memory_accessible(0x20000, 4, MemoryProtection::ReadWrite)); + + // 3) Resume: the store re-executes and the program returns. + match instance.run().unwrap() { + InterruptKind::Finished => {} + other => panic!("expected finished, got {other:?}"), + } + + // 4) The stored value is now visible. + assert_eq!(instance.read_memory(0x20000, 4).unwrap(), 0xAABB_CCDDu32.to_le_bytes()); + } + + #[test] + fn hv_static_oob() { + if hypervisor_unavailable() { + return; + } + // Static paging: a store to an unmapped address faults -> Trap (matches generic; + // Segfault is only produced with dynamic paging). + let (interrupt, _i) = run_hv(&[asm::store_imm_indirect_u32(A0, 0, 0x1234), asm::trap()], &[(A0, 0x0100_0000)]); + assert!(matches!(interrupt, InterruptKind::Trap), "unexpected interrupt: {interrupt:?}"); + } + + #[test] + fn hv_host_calls() { + if hypervisor_unavailable() { + return; + } + // A small "real" program: two host calls with register round-trips and arithmetic + // across the resulting basic blocks, then a return to the host. + // ecalli(0); A0 += 1; ecalli(0); A0 += 1; ret + let _ = env_logger::try_init(); + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest32); + builder.add_export_by_basic_block(0, b"main"); + builder.add_import(b"hostfn"); + builder.set_code( + &[ + asm::ecalli(0), + asm::add_imm_32(A0, A0, 1), + asm::ecalli(0), + asm::add_imm_32(A0, A0, 1), + asm::ret(), + ], + &[], + ); + let blob = ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap(); + + let mut config = Config::default(); + config.set_allow_experimental(true); + config.set_backend(Some(BackendKind::Compiler)); + config.set_sandbox(Some(crate::SandboxKind::Hypervisor)); + let engine = Engine::new(&config).unwrap(); + let mut module_config = ModuleConfig::default(); + module_config.set_page_size(4096); + let module = Module::from_blob(&engine, &module_config, blob).unwrap(); + let entry = module.exports().find(|e| e == "main").unwrap().program_counter(); + + let mut instance = module.instantiate().unwrap(); + instance.set_reg(Reg::RA, crate::RETURN_TO_HOST); + instance.set_reg(A0, 0); + instance.set_next_program_counter(entry); + + // First host call: the host provides a base value in A0. + assert!(matches!(instance.run().unwrap(), InterruptKind::Ecalli(0))); + instance.set_reg(A0, 10); + // Resume -> A0 += 1 -> second host call. + assert!(matches!(instance.run().unwrap(), InterruptKind::Ecalli(0))); + assert_eq!(instance.reg(A0), 11); // host edit + arithmetic survived the round-trip + instance.set_reg(A0, 20); + // Resume -> A0 += 1 -> ret. + assert!(matches!(instance.run().unwrap(), InterruptKind::Finished)); + assert_eq!(instance.reg(A0), 21); + } +} diff --git a/fuzz/Cargo.lock b/fuzz/Cargo.lock index 8576480bf..e0c95253b 100644 --- a/fuzz/Cargo.lock +++ b/fuzz/Cargo.lock @@ -261,6 +261,7 @@ name = "polkavm-fuzz" version = "0.0.0" dependencies = [ "arbitrary", + "libc", "libfuzzer-sys", "polkavm", "polkavm-common", diff --git a/fuzz/Cargo.toml b/fuzz/Cargo.toml index b37c3fe43..7e273676f 100644 --- a/fuzz/Cargo.toml +++ b/fuzz/Cargo.toml @@ -10,10 +10,12 @@ cargo-fuzz = true [dependencies] arbitrary = { version = "1.4.2", features = ["derive"] } libfuzzer-sys = "0.4" +libc = "0.2" [dependencies.polkavm] path = "../crates/polkavm" -features = ["export-internals-for-testing"] +# `generic-sandbox` runs the recompiler where the zygote sandbox is absent (macOS, AArch64); unused on x86-64 Linux. +features = ["export-internals-for-testing", "generic-sandbox"] [dependencies.polkavm-linker] path = "../crates/polkavm-linker" diff --git a/fuzz/fuzz_targets/fuzz_polkavm.rs b/fuzz/fuzz_targets/fuzz_polkavm.rs index 85e5fa30b..b738ec205 100644 --- a/fuzz/fuzz_targets/fuzz_polkavm.rs +++ b/fuzz/fuzz_targets/fuzz_polkavm.rs @@ -524,12 +524,35 @@ fn transform_code(data: Vec) -> Vec { fn build_program_blob(data: Vec) -> ProgramBlob { let code = transform_code(data); + if std::env::var("DEBUG_FUZZ_PROG").is_ok() { + eprintln!("[prog] {} instructions:", code.len()); + for inst in &code { + eprintln!(" {inst:?}"); + } + } + let mut builder = ProgramBlobBuilder::new(InstructionSetKind::Latest64); builder.add_export_by_basic_block(0, b"main"); builder.set_code(&code, &[]); ProgramBlob::parse(builder.into_vec().unwrap().into()).unwrap() } +fn native_page_size() -> u32 { + // SAFETY: `sysconf` has no preconditions and `_SC_PAGESIZE` always resolves to a positive value. + let size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) }; + u32::try_from(size).unwrap_or(4096) +} + +/// Selects the recompiler; enables the experimental generic sandbox off x86-64 Linux (e.g. macOS/AArch64). +fn configure_recompiler(config: &mut polkavm::Config) { + config.set_backend(Some(polkavm::BackendKind::Compiler)); + #[cfg(not(all(target_arch = "x86_64", target_os = "linux")))] + { + config.set_allow_experimental(true); + config.set_sandbox(Some(polkavm::SandboxKind::Generic)); + } +} + fn interpreter_fuzzer_harness(data: Vec, cache_config: Option) { let blob = build_program_blob(data); @@ -540,6 +563,7 @@ fn interpreter_fuzzer_harness(data: Vec, cache_config: Option) { let mut module_config = ModuleConfig::default(); module_config.set_strict(false); + // Both backends must share a page size so their memory maps (and thus the comparison) agree. + module_config.set_page_size(native_page_size()); module_config.set_gas_metering(Some(polkavm::GasMeteringKind::Sync)); let module = polkavm::Module::from_blob(&engine, &module_config, blob.clone()).unwrap(); @@ -582,12 +608,13 @@ fn correctness_fuzzer_harness(data: Vec) { let mut instance_recompiler = { let mut config = polkavm::Config::new(); - config.set_backend(Some(polkavm::BackendKind::Compiler)); + configure_recompiler(&mut config); let engine = Engine::new(&config).unwrap(); let mut module_config = ModuleConfig::default(); module_config.set_strict(false); + module_config.set_page_size(native_page_size()); module_config.set_gas_metering(Some(polkavm::GasMeteringKind::Sync)); let module = polkavm::Module::from_blob(&engine, &module_config, blob.clone()).unwrap();