feat: shadow fuzzer integration for ream

[Sahilgill24]

What was wrong?

This PR is for integration of ream with the lean-shadow-fuzzer

How was it fixed?

WIP

Current Progress

• As mentioned in the issue, shadow does not work with custom heap allocator's such as jemalloc, so made it optional and it cannot compile with shadow-integration feature flag

• under the shadow-integration feature flag, set the runtime to single threaded instead of multi threaded.

To-Do

• I have read CONTRIBUTING.md.
• This PR title follows conventional commits.

[KolbyML]

Ream's shadow integration will be done internally sorry for the inconvince

[KolbyML]

myb

[Sahilgill24]

Current Status

I just wanted to share the current status of this PR as well as an implementation dilemma here

• As mentioned in this shadow issue, shadow does not work with custom heap allocator's such as jemalloc, so made it optional and it cannot compile with shadow-integration feature flag.
• under the shadow-integration feature flag, set the runtime to single threaded instead of multi threaded.
• ported the quinn-udp crate patch from ethlambda's implementation to ours, as both use quinn-udp and GSO/GRO based syscalls used by the crate are not emulated in Shadow. As the crate is small I am also trying to make these changes feature gates directly inside quinn-udp crate itself also.
• added the sim cost module to model the proof aggregation/merge/verification cost without actually proving.

Regarding Fake XMSS proofs

Clients that are integrated with the shadow fuzzer currently use fake dummy XMSS proofs and model the CPU cost for the aggregation steps, ethlambda and zeam both use deterministic pseudo random number generation(PRNG) methods to generate random bytes to model this proof and under the shadow feature use it for all aggregation/merge/verification tasks. They can do this because their whole XMSS-aggregation API is bytes-in, bytes-out:

  // ethlambda 
  pub fn aggregate_signatures(
      public_keys: Vec<ValidatorPublicKey>,
      signatures: Vec<ValidatorSignature>,
      message: &H256,
      slot: u32,
  ) -> Result<ByteList512KiB, AggregationError>;

  pub fn merge_type_1s_into_type_2(
      type_1s: Vec<(Vec<ValidatorPublicKey>, ByteList512KiB)>,
  ) -> Result<ByteList512KiB, AggregationError>;

  pub fn split_type_2_by_message(
      proof_data: &[u8],
      pubkeys_per_component: Vec<Vec<ValidatorPublicKey>>,
      message: &H256,
  ) -> Result<ByteList512KiB, AggregationError>;

as the return type is just bytes, under shadow-integration they can hence return a raw PRNG blob(bytes) directly seeded from (message, slot, each child's proof bytes, count)

but in ream the type_2.rs API instead passes and returns native objects, not bytes:

// ream 
pub fn type_1_aggregate(
    children: &[SingleMessageAggregate],
    raw_xmss: &[(PublicKey, Signature)],
    message: &[u8; 32],
    slot: u32,
) -> Result<SingleMessageAggregate>;
pub fn type_2_merge(parts: Vec<SingleMessageAggregate>) -> Result<MultiMessageAggregate>;
pub fn type_2_split(proof: MultiMessageAggregate, index: usize) -> Result<SingleMessageAggregate>;
pub fn type_1_verify(proof: &SingleMessageAggregate) -> Result<()>;
pub fn type_2_verify(proof: &MultiMessageAggregate) -> Result<()>;

although we can compress an object to bytes (_to_wire), but reversing it (_from_wire → decompress_without_pubkeys) runs validation's which random bytes can't directly pass, and SingleMessageAggregate / MultiMessageAggregate also can't be constructed directly from scratch as their inner Proof fields are private. So the PRNG based approach cannot be directly ported to ream.

Now we can either change type_2.rs API to bytes-in/bytes-out which would also need changes at every call site, or as it is only a placeholder dummy we can embed a real proof captured offline as a prototype and just using it along with the (message, slot) to create a valid object.

For the demo below I used the second option, and the fuzzer runs cleanly for ream locally.

one kind of a difference in the second approach is that child_bytes are not used: ethlambda's PRNG seed folds in every child's proof bytes (the for (_, proof) in &children { parts.push(proof …) } above), so its dummy varies with the exact aggregation input. the ream approach only seeds determinism only from (message, slot) — the prototype's Proof<F> payload is copied directly from the prototype, as we are not doing some actual verification I think this approach can be used. I wanted a confirmation on this from you all

Demo

With changes to the fuzzer on my local for ream, the fuzzer integration is working completely fine and matching the behaviour of other clients as well.

you can watch the short Demo here
(git won't allow files larger than 10MB :( )

[Kayden-ML]

Hey @Sahilgill24 great work so far I was wondering when you have time if you could create a pr adding ream support in the https://github.com/kamilsa/lean-shadow-fuzzer repo and can you create a readme file in ream explaining how to run ream using shadow

[Sahilgill24]

Thanks @Kayden-ML, I just wanted to confirm one or two things in today's meet once.
I have all the required changes already completed on my local, would just have to make the pr afterwards :)