Skip to content

chore(amber): move test-only deps to dev-requirements.txt#5692

Merged
Yicong-Huang merged 3 commits into
apache:mainfrom
Yicong-Huang:chore/move-pytest-to-dev-requirements
Jun 15, 2026
Merged

chore(amber): move test-only deps to dev-requirements.txt#5692
Yicong-Huang merged 3 commits into
apache:mainfrom
Yicong-Huang:chore/move-pytest-to-dev-requirements

Conversation

@Yicong-Huang

@Yicong-Huang Yicong-Huang commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Four test-only pins were sitting in the wrong file (amber/requirements.txt), which leaked them into the production Docker image and the LICENSE-binary snapshot. This PR moves them.

Pin Move
pytest==7.4.0 requirements.txtdev-requirements.txt
pytest-reraise==2.1.2 requirements.txtdev-requirements.txt
pytest-timeout==2.2.0 requirements.txtdev-requirements.txt
iniconfig==1.1.1 requirements.txt → dropped entirely (pytest transitive, auto-installed)

Side effects:

File Change
amber/system-requirements-lock.txt drop the same 4 lines (header: "This file must be updated whenever requirements.txt or operator-requirements.txt changes")
amber/LICENSE-binary-python drop the same 4 entries from the BSD 2-Clause / MIT blocks

Why this is safe

Consumer Before After
CI pyamber / amber-integration stacks install requirements.txt + operator-requirements.txt + dev-requirements.txt — get pytest twice install all three — get pytest once (from dev-requirements)
bin/computing-unit-master.dockerfile (prod) installs requirements.txt + operator-requirements.txt — gets pytest, doesn't need it installs same two — no longer pulls pytest
bin/computing-unit-worker.dockerfile (prod) same as master same as master
PveManager.createNewPve (per-user PVEs) installs requirements.txt only — gets pytest, doesn't need it installs same — no longer pulls pytest
check_binary_deps.py (LICENSE drift gate) requires LICENSE-binary entries because requirements.txt declares them no longer requires them; entries removed in lockstep

The 1 non-test pytest import I found (amber/src/main/scala/.../aiassistant/test_type_annotation_visitor.py) is a pytest test that's accidentally placed under src/main by filename convention (test_*); it's not invoked by the production runtime.

Any related issues, documentation, discussions?

Closes #5691

How was this PR tested?

  • git diff upstream/main --stat — confirms only the four files above are touched, with the expected +6/-12 shape
  • grep -ciE "^(pytest|iniconfig| - pytest| - iniconfig)" amber/requirements.txt amber/system-requirements-lock.txt amber/LICENSE-binary-python — all three return 0 after the change
  • Repo-wide grep -rIE "^(from|import)[[:space:]]+pytest" shows 48 usages in amber/src/test/python (CI installs dev-requirements.txt, so they keep working) and the 1 non-runtime file noted above

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.7 [1M context])

pytest, pytest-reraise, pytest-timeout, and iniconfig were pinned in
amber/requirements.txt despite being test-only. Per the header comment
in amber/dev-requirements.txt, test deps belong there so they stay out
of the LICENSE-binary snapshot and the production CU Docker images
(which install requirements.txt + operator-requirements.txt only).

Move pytest, pytest-reraise, pytest-timeout to dev-requirements.txt.
Drop iniconfig entirely — it is pytest's transitive and gets pulled
in automatically.

Remove matching entries from amber/system-requirements-lock.txt (its
header says it tracks requirements + operator-requirements) and from
amber/LICENSE-binary-python.

Closes apache#5691
@github-actions github-actions Bot added dependencies Pull requests that update a dependency file pyamber labels Jun 13, 2026
@codecov-commenter

codecov-commenter commented Jun 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.82%. Comparing base (a044287) to head (4341333).
⚠️ Report is 14 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5692      +/-   ##
============================================
+ Coverage     52.79%   52.82%   +0.02%     
  Complexity     2548     2548              
============================================
  Files          1090     1090              
  Lines         42150    42175      +25     
  Branches       4529     4529              
============================================
+ Hits          22253    22278      +25     
  Misses        18575    18575              
  Partials       1322     1322              
Flag Coverage Δ *Carryforward flag
access-control-service 71.42% <ø> (ø) Carriedforward from 67451db
agent-service 34.36% <ø> (ø) Carriedforward from 67451db
amber 52.71% <ø> (ø) Carriedforward from 67451db
computing-unit-managing-service 1.65% <ø> (ø) Carriedforward from 67451db
config-service 56.71% <ø> (ø) Carriedforward from 67451db
file-service 57.06% <ø> (ø) Carriedforward from 67451db
frontend 47.93% <ø> (ø) Carriedforward from 67451db
pyamber 89.72% <ø> (-0.95%) ⬇️
python 90.74% <ø> (ø) Carriedforward from 67451db
workflow-compiling-service 58.69% <ø> (ø) Carriedforward from 67451db

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 5 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main a044287 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 387 0.236 25,078/34,831/34,831 us 🔴 +16.4% / 🔴 -5.9%
bs=100 sw=10 sl=64 814 0.497 120,596/157,608/157,608 us ⚪ within ±5% / 🔴 +12.7%
bs=1000 sw=10 sl=64 932 0.569 1,073,642/1,105,879/1,105,879 us ⚪ within ±5% / 🔴 -10.5%
Baseline details

Latest main a044287 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 387 tuples/sec 440 tuples/sec 410.82 tuples/sec -12.0% -5.8%
bs=10 sw=10 sl=64 MB/s 0.236 MB/s 0.268 MB/s 0.251 MB/s -11.9% -5.9%
bs=10 sw=10 sl=64 p50 25,078 us 21,553 us 23,785 us +16.4% +5.4%
bs=10 sw=10 sl=64 p95 34,831 us 32,793 us 34,980 us +6.2% -0.4%
bs=10 sw=10 sl=64 p99 34,831 us 32,793 us 34,980 us +6.2% -0.4%
bs=100 sw=10 sl=64 throughput 814 tuples/sec 811 tuples/sec 891.94 tuples/sec +0.4% -8.7%
bs=100 sw=10 sl=64 MB/s 0.497 MB/s 0.495 MB/s 0.544 MB/s +0.4% -8.7%
bs=100 sw=10 sl=64 p50 120,596 us 119,826 us 112,277 us +0.6% +7.4%
bs=100 sw=10 sl=64 p95 157,608 us 158,948 us 139,802 us -0.8% +12.7%
bs=100 sw=10 sl=64 p99 157,608 us 158,948 us 139,802 us -0.8% +12.7%
bs=1000 sw=10 sl=64 throughput 932 tuples/sec 939 tuples/sec 1,041 tuples/sec -0.7% -10.5%
bs=1000 sw=10 sl=64 MB/s 0.569 MB/s 0.573 MB/s 0.635 MB/s -0.7% -10.4%
bs=1000 sw=10 sl=64 p50 1,073,642 us 1,062,359 us 972,714 us +1.1% +10.4%
bs=1000 sw=10 sl=64 p95 1,105,879 us 1,136,873 us 1,023,057 us -2.7% +8.1%
bs=1000 sw=10 sl=64 p99 1,105,879 us 1,136,873 us 1,023,057 us -2.7% +8.1%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,516.22,200,128000,387,0.236,25077.55,34830.96,34830.96
1,100,10,64,20,2455.50,2000,1280000,814,0.497,120596.39,157607.74,157607.74
2,1000,10,64,20,21467.66,20000,12800000,932,0.569,1073642.25,1105879.20,1105879.20

pluggy is a pytest transitive (not directly imported anywhere in the
repo); once pytest moves to dev-requirements.txt it is no longer
installed during the LICENSE-binary snapshot, so check_binary_deps.py
flags it as stale. Drop the matching lines.
@Yicong-Huang Yicong-Huang requested a review from kunwp1 June 14, 2026 01:15
@Yicong-Huang

Copy link
Copy Markdown
Contributor Author

cc @SarahAsad23 PTAL

Comment thread amber/dev-requirements.txt Outdated

@kunwp1 kunwp1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Can you update the PR description because it's outdated with LoC changes and also include pluggy?

Per review discussion, transitive dependencies need not be declared even
in comments, so remove the iniconfig note from the pytest comment block.
@Yicong-Huang Yicong-Huang enabled auto-merge June 15, 2026 21:36
@Yicong-Huang

Copy link
Copy Markdown
Contributor Author

thanks, merging.

@Yicong-Huang Yicong-Huang added this pull request to the merge queue Jun 15, 2026
Merged via the queue into apache:main with commit 3dab771 Jun 15, 2026
25 checks passed
@Yicong-Huang Yicong-Huang deleted the chore/move-pytest-to-dev-requirements branch June 15, 2026 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file pyamber

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move test-only Python deps out of amber/requirements.txt

4 participants