Should this be usable without allocator_api?

[A0lson]

I initially attempted to use the acpi crate on a no_std system...

However, I found it is difficult to use (especially the master branch without the allocator_api feature being enabled.

Aside from a minor compile error, many of the table structs have all private fields and only seem usable by the allocator_api enabled code. (At least my initial interest was the MCFG and MADT tables).

Would there be any appetite for making all the low-level structs have public fields? Or is there some other design goal that I'm not aware of?

[IsaacWoods]

Hi, glad to hear you've tried the library out.

The refactor to the new allocator API was relatively recent and has not been pushed over the line in a form that is yet completely usable. There would definitely be appetite for making this possible, and if that requires making all the raw structures public then so be it - feel free to get something working in a fork and submitting a PR :)

Design wise, most users will still find it most useful to have an allocator available, and to let the library allocate freely and provide higher-level data structures. This is not yet easily done on main, so it might be better to use a released crates.io version for the time being, until I have time to polish main off. It's very typical to still provide heap allocation in a no_std context, but some people are interested in either not doing that, or making heap allocation fallable, hence the new direction.

Hopefully that answers some questions - lmk if not and if you've found any other pain points in using the libraries.

[SnowCheetos]

Ik this is super old but I want to comment on this specific topic.

After working extensively with Zig and nightly Rust’s allocator_api, my question has mostly become: why is this still not stable?
In systems code, the usual alternatives are fixed-capacity containers with MAX_* constants scattered all over, a global allocator that every subsystem implicitly shares (basically a ticking #UD or #GP bomb). Don't get me wrong, they all get the job done, but they are awkward when allocation domain and lifetime are part of correctness.
Here's a real example: I use a patched AML interpreter to discover firmware-described PS/2 and I²C/GPIO input devices. AML has to run early in boot, before the general kernel memory-management services exist, and I do not want it to depend on a global allocator.
allocator_api lets that interpreter use ordinary Vec, BTreeMap, Arc, strings, etc. while drawing every AML-owned allocation from one bounded arena:

unsafe impl Allocator<const N: usize> for AmlArena<N> { // ... }
const BYTES_FOR_AML = 2 << 20; // Now I only need to define this one constant
Interpreter<SomeHandler, AmlArena<BYTES_FOR_AML>> // Now I can sleep at night knowing that any overflow would mostly require just one capacity bump

The arena is the important part: one capacity defines the dynamic-memory budget for the AML path, its lifetime remains explicit, and malformed firmware can exhaust only that domain rather than silently consuming a global kernel heap. With fallible allocation at the relevant boundaries, failures are also much more self-contained.
Zig makes explicit allocator passing conventional in its standard library. Rust’s allocator-generic standard containers provide the same valuable model, but APIs such as Vec::new_in remain nightly-only under allocator_api. I think that is particularly valuable for this crate and for systems Rust generally.

Bit of an info dump; moral of my story is that I think allocator_api is particularly beneficial for this crate and systems programming in Rust.

[martin-hughes]

Why is this still not stable?

Well because we haven't done it yet 😉 More seriously - because Isaac and (more recently) me only have limited time due to work & family commitments and it hasn't been useful for us, personally.

Per the notes in #306, both Isaac and me are OK with the idea of allowing acpi to use the allocator_api - I think we both welcome your work on it @SnowCheetos 😊 Personally, I think it would be a beneficial update to the crate.

Although I must point out that if the reason you want it to use a separate allocator is to keep it separate from other kernel memory, you could consider other methods of isolation that would be more robust (such as entirely separate processes).

To me, it seems that a big benefit of custom allocators (as compared to other isolation methods) would be more fine-grained control over caching and/or paging behaviour.

If anyone else picks this up in advance of @SnowCheetos, please take a look at their previous work and the notes in #306, there's plenty of good stuff there.

---

Some notes of my own - hopefully interesting questions for your own hobby OS. I want to emphasise that your proposal has value, these are just me doing a bit of bikeshedding:

First though - some good news! Allocator API seems to be progressing again: rust-lang/rust#156882

basically a ticking #UD or #GP bomb

Why? Especially why #UD? Presumably splitting allocations between allocators makes this problem worse as you've added the possibility of the wrong allocator being called to release memory.

allocation domain [...is...] part of correctness

Yet if there's only one domain (the global allocator), presumably allocation domain can never contribute to incorrectness? Whereas with multiple allocation domains you could accidentally introduce incorrect or unsound behaviour.

AML has to run early in boot, before the general kernel memory-management services exist

Does it? That feels like a design choice you could alter... My most recent kernel takes a memory map from the firmware and starts the global allocator before using any AML. Alternatively, if you're getting something more advanced from ACPI such as NUMA, you could start with a "startup allocator" that is consumed/replaced by the "main allocator" once the system/ACPI is more fully running.

I'd be curious to see your OS some more if you're open to sharing it - I always learn something from other people's code!

I do not want it to depend on a global allocator

Hence your PR I suppose! Although see my next bit of bikeshedding - there are ways to encapsulate acpi so that it has its own allocator and yet still has access to ring 0 / supervisor mode.

The arena is the important part: one capacity defines the dynamic-memory budget for the AML path, its lifetime remains explicit, and malformed firmware can exhaust only that domain rather than silently consuming a global kernel heap.

To me, this sounds a bit like encapsulating acpi in its own process... Obviously microkernels come with their own tradeoffs! But you could do a sort of hybrid where the acpi process is a separate executable that also runs in ring 0 (or supervisor mode) for speedy access to the rest of the kernel - then it could have its own "global" allocator that's separate from the main system allocator. It just becomes a linking / loading problem of how to get that acpi process into the system at startup. (Perhaps via an initrd?)

[SnowCheetos]

Why is this still not stable?

Well because we haven't done it yet 😉 More seriously - because Isaac and (more recently) me only have limited time due to work & family commitments and it hasn't been useful for us, personally.

Per the notes in #306, both Isaac and me are OK with the idea of allowing acpi to use the allocator_api - I think we both welcome your work on it @SnowCheetos 😊 Personally, I think it would be a beneficial update to the crate.

Although I must point out that if the reason you want it to use a separate allocator is to keep it separate from other kernel memory, you could consider other methods of isolation that would be more robust (such as entirely separate processes).

To me, it seems that a big benefit of custom allocators (as compared to other isolation methods) would be more fine-grained control over caching and/or paging behaviour.

If anyone else picks this up in advance of @SnowCheetos, please take a look at their previous work and the notes in #306, there's plenty of good stuff there.

Some notes of my own - hopefully interesting questions for your own hobby OS. I want to emphasise that your proposal has value, these are just me doing a bit of bikeshedding:

First though - some good news! Allocator API seems to be progressing again: rust-lang/rust#156882

basically a ticking #UD or #GP bomb

Why? Especially why #UD? Presumably splitting allocations between allocators makes this problem worse as you've added the possibility of the wrong allocator being called to release memory.

allocation domain [...is...] part of correctness

Yet if there's only one domain (the global allocator), presumably allocation domain can never contribute to incorrectness? Whereas with multiple allocation domains you could accidentally introduce incorrect or unsound behaviour.

AML has to run early in boot, before the general kernel memory-management services exist

Does it? That feels like a design choice you could alter... My most recent kernel takes a memory map from the firmware and starts the global allocator before using any AML. Alternatively, if you're getting something more advanced from ACPI such as NUMA, you could start with a "startup allocator" that is consumed/replaced by the "main allocator" once the system/ACPI is more fully running.

I'd be curious to see your OS some more if you're open to sharing it - I always learn something from other people's code!

I do not want it to depend on a global allocator

Hence your PR I suppose! Although see my next bit of bikeshedding - there are ways to encapsulate acpi so that it has its own allocator and yet still has access to ring 0 / supervisor mode.

The arena is the important part: one capacity defines the dynamic-memory budget for the AML path, its lifetime remains explicit, and malformed firmware can exhaust only that domain rather than silently consuming a global kernel heap.

To me, this sounds a bit like encapsulating acpi in its own process... Obviously microkernels come with their own tradeoffs! But you could do a sort of hybrid where the acpi process is a separate executable that also runs in ring 0 (or supervisor mode) for speedy access to the rest of the kernel - then it could have its own "global" allocator that's separate from the main system allocator. It just becomes a linking / loading problem of how to get that acpi process into the system at startup. (Perhaps via an initrd?)

Ha, impressive how spot on you are in the speculation merely from my sloppy comments. I should clarify first tho

Well because we haven't done it yet 😉 More seriously - because Isaac and (more recently) me only have limited time due to work & family commitments and it hasn't been useful for us, personally.

Absolutely understandable! Thank you guys for making this crate to begin with (I still remembered the day when I scrolled through crates.io and found acpi, immediately - "oh well seems like I wasted a whole bunch of time hand rolling a much worse version"), it collectively probably saved me hundreds of hours; my complaints on the allocator_api was mainly targeting Rust itself.

Why? Especially why #UD? Presumably splitting allocations between allocators makes this problem worse as you've added the possibility of the wrong allocator being called to release memory.

Yeah I can see how it reads like nonsense without any context (also I made a typo there, I meant to say #PF and #GP); but it ties to this one point:

AML has to run early in boot, before the general kernel memory-management services exist

Without dragging in details, the OS I'm working on relies on an analytically proven DAG model to make certain runtime security guarantees, provided that certain restrictions/conditions are satisfied. To summarize, capabilities are opaque fd-like objects (so 'everything is a capability') which carry kernel stamped cryptographic tokens. At first glance it might seem that the provence step to be a major bottleneck, and indeed it would ordinarily, but given one more initial assumption - DAG roots are static after boot , two important mathematical properties can be exploited:

1. Instead of keeping a DAG model/cap table structure, we can compute the transitive closure via affine transformations in O(1). Essentially, we can determine whether node A is reachable to node B in O(1); now comes the fun part, add in one more definition - derive which attenuates existing capabilities, you can now see how the entire system, anywhere from PCI functions to DMA buffers, could be derived from a set of static DAG roots (which is most intuitively modeled as anything 'physical').
2. This hands the entire hosted runtime a major security guarantee (assuming no bugs, ofc...) - no new nodes can be created out of thin air, any new nodes must have been derived from one of the root nodes, which the kernel is the owner of. Any resource access that userland could mathematically have at any moment must have been explicitly derived and attenuated by someone who had that access to begin with; and to bring it closer to home, it eliminates one nasty class of bugs (assisted by rust lifetimes) - UAF

The catch? We cannot (at least haven't successfully) prove that the guarantees hold when the DAG roots are dynamically scaled. Doesn't mean that we cannot model dynamic hardware topology, not doing so would make the OS pretty useless. It does, however, shift some of the weight to the definition of the dag roots themselves.

Second catch: I can't possibly know exactly what resources are present before compilation (not if I want to boot it on any other machines), meaning that some sort of dynamic allocation is required; yet to satisfy the DAG model, the roots must be frozen post initialization and discovery.

Third and final catch: memory itself is modeled as capabilities, meaning during bootstrap we have cannot guarantee a safe global heap. But in the code itself I'd still want to use ergonomic apis like core::alloc::vec or string, hmm... only if I could plug in a dedicated boot time Allocator, which can be further improved to be append-only and freeze-able/fallible. See why I am obsessed with allocator_api now 😅 ?

This now finally ties back to ACPI and AML; since I need to enumerate the MCFG tables to record PCI devices, and we'd want to keep PCI (technically the RC would be the root, same difference) as roots, they need to stay immutable post init, and the way I chose achieve that was to evaluate whatever AML methods needed during bootstrap, when a global heap is absent. For rather obvious reasons I chose not to 'fake' a global allocator with a bump arena or smth, that meant I either: A) refactor the AML module to be allocator agnostic; B) hand roll my own AML interpreter. The choice was pretty clear there lol.

Hope this gave some clarity in my rationale behind my comments and PRs. Love to share/discuss more if needed.

[martin-hughes]

Ah cool! Thanks for that summary. I've never tried using or working on a capabilities-based OS, although I like the concept. I should probably try one one day.

Is your OS public? I'd love to take a look...

[SnowCheetos]

I would love for some external reviews too! It's private rn but I suppose external collaborator could work?