PoC SHM Transport

[kurtsiegfried]

I started working on a shared memory transport implementation for clients that want lower latency / higher throughput and can live on the same host as the server process: kurtsiegfried#2

Currently it works as a pair of SPSC ringbuffers between client and server, with a unix socket for setting up the connection. In the future, it should be possible to have an MPSC structure to reduce some of the client overhead.

This is quite a large changeset, and not fully baked, but I'm curious if this is something that's open to being accepted down the road.

[hubcio]

this is very interesting! thank you for this.

have you considered using some rust crate that'll handle IPC? we actually wanted to have IPC for example for cases where connectors runtime is running on same host as iggy server.

i went through thecode on your fork, few questions:

1. iceoryx2 is apache-2/mit and does zero-copy shm with request-response. your core/shm pulls only libc, so hand-rolling was deliberate. what did your own ring buy you? the stack is close to 9k lines and the ring is the part we'd have to keep correct forever.

2. did you try a plain unix domain socket with the existing tcp framing first? that would show how much of the win is "no tcp" vs "no syscalls". also the numbers are macos vs tcp loopback, we run on linux with io_uring, so linux numbers would matter a lot more.

3. it's still copy-in, copy-out as far as i can tell. is zero-copy on the table at all, or is the endgame fewer syscalls?

4. 48 MiB per connection, 1.5 GiB at max_connections=32. why did you choose 8 MiB regions as the default?

5. in a cluster, when the partition primary sits on another node, what does the client do today?

overall, i'm rly excited about what you did. nice.

[kurtsiegfried]

1. I saw someone answered my question on Reddit mentioning iceoryx2 as well. I wasn't happy about pulling in another entire project for an experiment, and when I initially looked at it a few years ago, it could only handle fixed size messages (no longer true). Agreed that it's a large piece of code that's critical to the transport, and worth choosing deliberately.

2. I didn't try the socket route, my experience with messaging protocols has left me with a distrust of TCP in low latency domains. On the Linux front, I have a machine that I need to get cabled back up, and will do some additional testing.

3. Correct for now. ZC should be achievable at least on the inbound side, it's something I plan to work on.

4. I wanted clients to be able to send large messages without thinking too much about it as a default. In a low latency deployment, I imagine the messages are going to be significantly smaller, and able to be scaled down.

5. In short, currently just raises an error to the client. It seemed like an ok-ish decision for now, with some options for having the server relay messages to the primary, or redirecting to a different transport more explicitly down the road.

Thanks for taking a look, and the thoughtful questions.

[spetz]

IMO this is a great idea, and the current results are already very promising. BTW, I did use Aeron back in the day, glad to see someone who's experienced in it contributing to Iggy! :)

[kurtsiegfried]

Yeah, I use it in my current trading system along with Kafka for iceberg persistence. My original premise for starting this was an exploration of whether I could use a single system for local transport + remote iceberg persistence. Either way, it's been a fun way to get to know the codebase.