feat(hydro_aws): add AWS crate, SQS integration by MingweiSamuel · Pull Request #2746 · hydro-project/hydro

MingweiSamuel · 2026-04-15T23:41:01Z

Not sure if this should land in hydro_std or a new e.g. hydro_aws (or even finer grain hydro_sqs) crate

cloudflare-workers-and-pages · 2026-04-15T23:41:26Z

Deploying hydro with Cloudflare Pages

Latest commit:	`c581e9d`
Status:	✅ Deploy successful!
Preview URL:	https://628c4b35.hydroflow.pages.dev
Branch Preview URL:	https://mingwei-sqs-rebase.hydroflow.pages.dev

View logs

Copilot

Pull request overview

Adds an AWS + SQS integration example to hydro_test, gated behind new Cargo features, to demonstrate sourcing from and writing to SQS within Hydro flows.

Changes:

Introduce hydro_test::aws module with helpers to load AWS SDK config and build an SQS client.
Add SQS source/sink utilities and a runnable aws_sqs example.
Add feature flags (aws, aws_sqs) and wire the example to required-features.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
hydro_test/src/lib.rs	Gate new `aws` module behind `feature = "aws"`.
hydro_test/src/aws/mod.rs	Add AWS SDK config source helper and stageleft rewrite mappings.
hydro_test/src/aws/sqs.rs	Add SQS client/source/dest utilities.
hydro_test/examples/aws_sqs.rs	Add deployment-backed example using the new SQS helpers.
hydro_test/Cargo.toml	Add `aws`/`aws_sqs` features, optional AWS deps, and example feature gating.
Cargo.lock	Pull in AWS SDK dependency graph.
.vscode/settings.json	Enable `aws_sqs` feature for rust-analyzer by default.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run>

shadaj · 2026-04-22T00:45:07Z

+        .map(q!(|(message, client)| self::sqs_send(
+            &client, queue_url, message
+        )))
+        .resolve_futures_blocking();


We should probably split out APIs for side effecting resolve vs non side effecting.

Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run>

- Add hydro_test/src/kafka/mod.rs with kafka_producer, kafka_consumer, dest_kafka, and setup_topic helpers following the SQS PR hydro-project#2746 pattern - Complete hydro_test/examples/kafka.rs: leader produces 1M financial transactions, consumer cluster computes per-account balances - Add 'kafka' feature flag gating rdkafka and futures-util as optional deps - Add llvm-tools component to rust-toolchain.toml for rust-lld

PR: #2746

- hydro_test/src/aws/sqs.rs: Replace redundant closures `|vec| futures_util::stream::iter(vec)` with direct function reference `futures_util::stream::iter` (lines 45 and 70) - hydro_test/examples/aws_sqs.rs: Replace explicit auto-deref `&*args.queue_url` with `&args.queue_url` (lines 63 and 80) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

- Add hydro_test/src/kafka/mod.rs with kafka_producer, kafka_consumer, dest_kafka, and setup_topic helpers following the SQS PR hydro-project#2746 pattern - Complete hydro_test/examples/kafka.rs: leader produces 1M financial transactions, consumer cluster computes per-account balances - Add 'kafka' feature flag gating rdkafka and futures-util as optional deps - Add llvm-tools component to rust-toolchain.toml for rust-lld

PR: #2746

- hydro_test/src/aws/sqs.rs: Replace redundant closures `|vec| futures_util::stream::iter(vec)` with direct function reference `futures_util::stream::iter` (lines 45 and 70) - hydro_test/examples/aws_sqs.rs: Replace explicit auto-deref `&*args.queue_url` with `&args.queue_url` (lines 63 and 80) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

PR: #2746

Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

- Add hydro_test/src/kafka/mod.rs with kafka_producer, kafka_consumer, dest_kafka, and setup_topic helpers following the SQS PR hydro-project#2746 pattern - Complete hydro_test/examples/kafka.rs: leader produces 1M financial transactions, consumer cluster computes per-account balances - Add 'kafka' feature flag gating rdkafka and futures-util as optional deps - Add llvm-tools component to rust-toolchain.toml for rust-lld

Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

PR: #2746

- hydro_test/src/aws/sqs.rs: Replace redundant closures `|vec| futures_util::stream::iter(vec)` with direct function reference `futures_util::stream::iter` (lines 45 and 70) - hydro_test/examples/aws_sqs.rs: Replace explicit auto-deref `&*args.queue_url` with `&args.queue_url` (lines 63 and 80) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

PR: #2746

Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

PR: #2746

PR: #2746 SQS working! PR: #2746 cleanup sqs one-write code PR: #2746 enable credentials-login feature for getting login from `aws` cli PR: #2746 enable credentials-login feature for getting login from `aws` cli PR: #2746 Fix clippy warnings in hydro_test SQS module - hydro_test/src/aws/sqs.rs: Replace redundant closures `|vec| futures_util::stream::iter(vec)` with direct function reference `futures_util::stream::iter` (lines 45 and 70) - hydro_test/examples/aws_sqs.rs: Replace explicit auto-deref `&*args.queue_url` with `&args.queue_url` (lines 63 and 80) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746 re-enable aws feature gating PR: #2746 address PR #2746 review comments for AWS SQS support Changes made: - Fix typo: `chaning` → `changing` in sqs.rs comment - Replace concrete AWS account ID/queue URL with placeholders in example - Weaken `source_sqs_fifo` guarantees from `TotalOrder`/`ExactlyOnce` to `NoOrder`/`AtLeastOnce` since SQS FIFO only guarantees ordering within a single message group; removed `NonDet` parameter; added detailed doc comment - Add doc comment to `dest_sqs` noting it's for standard queues only (no `message_group_id` set) - Add `features = ["alloc"]` to `futures-util` dependency for stream module - Remove `aws_sqs` from `.vscode/settings.json` cargo features - Improve `nondet!` justification comments in example - Remove unused `TotalOrder` import from sqs.rs Comments dismissed as incorrect/not actionable: - EC2 hardcoded region/AMI: acceptable for an example - Delete-before-return: already has TODO, AtLeastOnce marker is accurate - `?` terminates stream on empty poll: intentional design (same as resolved) - flat_map yields Vec not Message: wrong (Option iterates over inner Vec) Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

- Populated the hydro_aws crate with code from hydro_test/src/aws/ (mod.rs and sqs.rs), adding #[cfg(stageleft_runtime)] hydro_lang::setup!() and renaming the feature gate from "aws_sqs" to "sqs". - Configured hydro_aws/Cargo.toml as a published crate: publish = true, proper metadata (description, documentation, license, repository), features (sqs, stageleft_macro_entrypoint), dependencies (aws-config, aws-sdk-sqs, futures-util, ctor, hydro_lang, stageleft), build.rs with stageleft_tool::gen_final!(), and empty CHANGELOG.md. - Moved hydro_test/examples/aws_sqs.rs to hydro_aws/examples/aws_sqs.rs, updated imports to use hydro_aws directly, and changed TrybuildHost features to ["sqs"]. - Removed hydro_test/src/aws/ directory, all AWS features/dependencies from hydro_test/Cargo.toml, and the pub mod aws declaration from lib.rs. - Added hydro_aws to .github/workflows/release.yml explicit crate list and RELEASING.md example command. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

In aws-config 1.8.14, the `sso` feature (enabled by default) depended on `dep:ring`. Starting from 1.8.16, this was replaced with `dep:sha1`, eliminating the ring dependency entirely. Updated the version requirement from "1" to "1.8.16" to ensure ring is never pulled in. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #2746

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 74306c4 to 2fc30e0 Compare April 15, 2026 23:41

MingweiSamuel force-pushed the mingwei/sqs-rebase branch 3 times, most recently from 0f2fc28 to 780922c Compare April 20, 2026 18:39

MingweiSamuel changed the title ~~feat(dfir_lang): sqs rebase wip~~ feat(hydro_test): add example of AQS + SQS support Apr 20, 2026

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from ad30558 to 2ada126 Compare April 20, 2026 19:42

MingweiSamuel changed the title ~~feat(hydro_test): add example of AQS + SQS support~~ feat(hydro_test): add example of AWS + SQS support Apr 20, 2026

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 2ada126 to 67c7bb3 Compare April 20, 2026 19:43

MingweiSamuel marked this pull request as ready for review April 20, 2026 19:43

MingweiSamuel requested review from a team and Copilot April 20, 2026 19:43

Copilot started reviewing on behalf of MingweiSamuel April 20, 2026 19:43 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

shadaj approved these changes Apr 22, 2026

View reviewed changes

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from c1c9479 to 636a8b4 Compare April 22, 2026 23:04

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

add aws, sqs hydro_test

f7dd09c

PR: #2746

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

SQS working!

d7f5a17

PR: #2746

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

cleanup sqs one-write code

50143cf

PR: #2746

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

enable credentials-login feature for getting login from aws cli

016e66e

PR: #2746

MingweiSamuel added a commit that referenced this pull request Apr 28, 2026

enable credentials-login feature for getting login from aws cli

571ab07

PR: #2746

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 636a8b4 to 2befe7c Compare April 28, 2026 17:09

MingweiSamuel added a commit that referenced this pull request May 2, 2026

add aws, sqs hydro_test

2da08fe

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 2, 2026

SQS working!

8584f26

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 2, 2026

cleanup sqs one-write code

0a0e602

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 2, 2026

enable credentials-login feature for getting login from aws cli

fe2ea6e

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 2, 2026

enable credentials-login feature for getting login from aws cli

09d079c

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 2, 2026

re-enable aws feature gating

b32fb1a

PR: #2746

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 2df53f5 to 273f1f7 Compare May 2, 2026 05:06

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 273f1f7 to 8d8dd59 Compare May 5, 2026 20:18

MingweiSamuel added a commit that referenced this pull request May 8, 2026

add aws, sqs hydro_test

4ba9f34

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 8, 2026

SQS working!

7ba694b

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 8, 2026

cleanup sqs one-write code

46044f3

PR: #2746

MingweiSamuel force-pushed the mingwei/sqs-rebase branch from 8d8dd59 to 2a3bae6 Compare May 8, 2026 18:53

MingweiSamuel added a commit that referenced this pull request May 8, 2026

enable credentials-login feature for getting login from aws cli

d24952b

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 8, 2026

enable credentials-login feature for getting login from aws cli

7d3a11f

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 8, 2026

re-enable aws feature gating

7eefb84

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 12, 2026

add aws, sqs hydro_test

aa7a009

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 12, 2026

SQS working!

81df6b8

PR: #2746

MingweiSamuel added a commit that referenced this pull request May 12, 2026

cleanup sqs one-write code

5c80acb

PR: #2746

MingweiSamuel and others added 3 commits June 18, 2026 21:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(hydro_aws): add AWS crate, SQS integration#2746

feat(hydro_aws): add AWS crate, SQS integration#2746
MingweiSamuel wants to merge 3 commits into
mainfrom
mingwei/sqs-rebase

MingweiSamuel commented Apr 15, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shadaj Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

MingweiSamuel commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying hydro with Cloudflare Pages

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shadaj Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MingweiSamuel commented Apr 15, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Apr 15, 2026 •

edited

Loading