MIDRC-1292 Enable Nextflow tests in Kind CI + test range param + test PV ops by paulineribeyre · Pull Request #609 · uc-cdis/gen3-code-vigil

paulineribeyre · 2026-07-24T15:56:21Z

Link to JIRA ticket if there is one: https://ctds-planx.atlassian.net/browse/MIDRC-1292

Fixed in uc-cdis/gen3-workflow#169

New Features

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

github-actions · 2026-07-24T15:57:03Z

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

github-actions · 2026-07-24T16:02:28Z

Integration Tests

Failed to Prepare CI environment

Please find the Github Action logs here

github-actions · 2026-07-24T16:42:33Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_gen3_workflow.py	2	1	3
TOTAL	2	1	3

Test summary after rerunning failed integration tests

filepath	passed	SUBTOTAL
tests/test_gen3_workflow.py	1	1
TOTAL	1	1

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-07-24T16:50:56Z

Integration Tests on Kind

Test summary after running integration tests

filepath	passed	failed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	22	5	2	29
TOTAL	22	5	2	29

Test summary after rerunning failed integration tests

filepath	passed	SUBTOTAL
tests/test_gen3_workflow.py	5	5
TOTAL	5	5

github-actions · 2026-07-24T18:24:36Z

Integration Tests

filepath	passed	SUBTOTAL
tests/test_gen3_workflow.py	3	3
TOTAL	3	3

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2026-07-24T19:32:47Z

Integration Tests on Kind

Test summary after running integration tests

filepath	passed	failed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	1	1	1	3
TOTAL	1	1	1	3

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_gen3_workflow.py	1	1
TOTAL	1	1

…le-kind-nextflow

github-actions · 2026-07-24T23:10:21Z

Integration Tests

filepath	passed	SUBTOTAL
tests/test_gen3_workflow.py	3	3
TOTAL	3	3

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2026-07-25T00:30:22Z

Integration Tests on Kind

Test summary after running integration tests

filepath	passed	failed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	1	1	1	3
TOTAL	1	1	1	3

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_gen3_workflow.py	1	1
TOTAL	1	1

github-actions · 2026-07-27T16:51:34Z

Integration Tests on Kind

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	2	30
TOTAL	28	2	30

github-actions · 2026-07-27T17:26:49Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	29	0	1	30
tests/test_oauth2.py	15	0	0	15
tests/test_dicom_viewer.py	4	0	0	4
tests/test_centralized_auth.py	16	0	0	16
tests/test_graph_submit_and_query.py	10	3	1	14
tests/test_audit_service.py	3	0	3	6
tests/test_presigned_url.py	7	0	0	7
tests/test_requestor.py	6	0	0	6
tests/test_drs_endpoint.py	6	0	0	6
tests/test_data_upload.py	8	0	1	9
tests/test_dbgap.py	4	0	1	5
tests/test_user_token.py	5	0	0	5
tests/test_study_viewer.py	3	0	0	3
tests/test_user_data_library.py	2	0	2	4
tests/test_user_login_activation.py	2	0	1	3
tests/test_pfb_export.py	1	0	0	1
tests/test_login_page.py	2	0	0	2
tests/test_guppy_service.py	13	0	0	13
tests/test_discoverypage.py	1	0	0	1
tests/test_register_user.py	2	0	0	2
tests/test_client_credentials.py	1	0	0	1
tests/test_aggregate_mds.py	1	0	0	1
tests/test_gen3_sdk.py	1	0	0	1
tests/test_google_data_access.py	1	0	0	1
tests/test_gen3client.py	1	0	0	1
tests/test_manifest_service.py	1	0	0	1
tests/test_homepage.py	1	0	0	1
tests/test_oidc_client.py	2	0	0	2
tests/test_fence_admin.py	2	0	0	2
tests/test_ras_authn.py	0	0	3	3
tests/test_ras_passport.py	0	0	3	3
tests/test_gen3ff_landing_page.py	0	0	2	2
tests/test_env_sanity.py	0	0	1	1
tests/test_etl.py	0	0	1	1
tests/test_workspace.py	0	1	0	1
TOTAL	150	4	20	174

Test summary after rerunning failed integration tests

filepath	passed	failed	SUBTOTAL
tests/test_graph_submit_and_query.py	2	1	3
tests/test_workspace.py	1	0	1
TOTAL	3	1	4

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-07-27T17:52:51Z

Integration Tests on Kind

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	2	30
TOTAL	28	2	30

github-actions · 2026-07-27T18:01:26Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	27	2	1	30
TOTAL	27	2	1	30

Test summary after rerunning failed integration tests

filepath	passed	failed	SUBTOTAL
tests/test_gen3_workflow.py	1	1	2
TOTAL	1	1	2

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-07-30T05:58:39Z

Integration Tests on Kind

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	2	30
TOTAL	28	2	30

github-actions · 2026-07-30T22:49:49Z

Integration Tests on Kind

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	2	30
TOTAL	28	2	30

github-actions · 2026-07-30T22:54:30Z

Integration Tests

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	29	1	30
TOTAL	29	1	30

Please find the detailed integration test report here

Please find the Github Action logs here

…le-kind-nextflow

github-actions · 2026-08-03T19:06:16Z

Integration Tests on Kind

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	2	30
TOTAL	28	2	30

github-actions · 2026-08-03T19:09:03Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	1	1	30
TOTAL	28	1	1	30

Test summary after rerunning failed integration tests

filepath	passed	SUBTOTAL
tests/test_gen3_workflow.py	1	1
TOTAL	1	1

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-08-04T20:29:47Z

Integration Tests on Kind

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	2	30
TOTAL	28	2	30

github-actions · 2026-08-04T20:40:41Z

Integration Tests

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	29	1	30
TOTAL	29	1	30

Please find the detailed integration test report here

Please find the Github Action logs here

nss10

Some minor suggestions and questions. PR looks good overall.

nss10 · 2026-08-05T02:46:29Z

                else None
            )
-        logger.info(f"S3 {action.upper()} response:  {response}")
+        logger.debug(f"S3 {action.upper()} response:  {response}")


Thanks for reducing the clutter in logs. :)

nss10 · 2026-08-05T02:47:43Z

        user: str = "main_account",
        expected_status=200,
+        range: str = "",
    ):


Can we add a docstring with Args for this function? Rest of the params are self explanatory, but I don't think range is. And a docstring might be handy for future maintanence!

Also, can you keep expected_status as the last argument for the function, to remain consistent with all the other functions?

nss10 · 2026-08-05T15:04:46Z

+print(f"SKIP_BROKEN: {os.environ.get('SKIP_BROKEN')} => {SKIP_BROKEN}")
+
+
+# Test sequential file writes


I understand this file is sent to the TES pods to be executed as part of the test, but could you restructure the script to resemble a test suite?

Could each operation be moved into its own function, with all of the functions invoked at the end of the file? If one function fails, the script should continue running the remaining tests while logging which test failed.

I realize the current behavior is essentially binary—all of these tests fail with the S3 CSI driver and pass with S3Files—but organizing the script this way would make it much easier to add new test cases that we want to run on the pods in the future. It would also make the tests more descriptive and ensure that a single failing test does not prevent the remaining tests from running.

I think it's sufficiently self-explanatory and restructuring the script wouldn't help much

If the script keeps running if something fails, the TES task will fail to find the output files and will error anyway.

The ZIP test is dependent on the other 2 cases

I doubt we'll add many more PV posix tests

The assertions in the test itself would have to be restructured as well to keep checking other test cases even if a previous assertion fails. I don't think I should spend much more time on this PR if there is no great benefit

nss10 · 2026-08-05T15:10:25Z

+        Also a regression test for various persistent volumes issues caused by S3-CSI driver
+        limitations. Fixed by switching PVs to S3Files.
+        This is written as a single test to keep all the PV-related POSIX issues together, and to
+        run fewer TES tasks to speed up testing.


This is nice thinking :)

nss10 · 2026-08-05T15:14:00Z

+          expected offset 18154 but got 0" (Victoria's PLP workflow)
+          Note: unable to reproduce PDF, FastQ and GZ issues at this time.
+          MIDRC-1278 GZ file input: it works when the input is `s3://{s3_path_prefix}/input.txt.gz`.
+          We may be able to reproduce the issue by using a presigned URL instead, but that case is


This is true. I only noticed the issue when the gz file is received through presigned url.
DM from a thread with Sairam

Can you try using the gz files by uploading it to your S3 work directory temporarily? Once the ticket is resolved, you would be able to work with pre-signed urls directly.

nss10 · 2026-08-05T15:18:01Z

+        - Incremental/append writes to files already flushed to S3
+          Note: unable to reproduce issue at this time


This was happening while writing to log files. When testing victoria's workflow, the output log file only has the first few lines, but every append after that was silently ignored.

Were you not seeing any issue when you tested this 👇 in the script

gen3-code-vigil/gen3-integration-tests/test_data/gen3_workflow/test_pv_posix_ops.py

Lines 10 to 15 in f9f7151

# Test sequential file writes

with open("output.txt", "w") as f:

f.write("Initial sequential data\n")

if not SKIP_BROKEN:

with open("output.txt", "a") as f:

f.write("Second sequential data\n")

I did. I couldn't reproduce it initially and I think this comment might be out of date

Yeah, it is only reproduced while you are in a directory where the PV is mounted. It is possible that you might've tested in a non PV work dir.

For instance, the default workdir for this image quay.io/cdis/gen3-workflow:integration_tests_dicom_image is /utils, which is out of the PV boundary where sequential writes work just fine.

github-actions · 2026-08-07T16:59:40Z

Integration Tests on Kind

Test summary after running integration tests

filepath	passed	failed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	27	1	2	30
TOTAL	27	1	2	30

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_gen3_workflow.py	1	1
TOTAL	1	1

github-actions · 2026-08-07T17:53:56Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	1	1	30
TOTAL	28	1	1	30

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_gen3_workflow.py	1	1
TOTAL	1	1

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-08-10T15:51:05Z

Integration Tests on Kind

filepath	passed	SUBTOTAL
tests/test_gen3_workflow.py	1	1
TOTAL	1	1

github-actions · 2026-08-10T15:54:21Z

Integration Tests

filepath	passed	SUBTOTAL
tests/test_gen3_workflow.py	1	1
TOTAL	1	1

Please find the detailed integration test report here

Please find the Github Action logs here

paulineribeyre added 6 commits June 10, 2026 14:00

test_s3_download_range_parameter

28ed84d

merge tests

695fac1

test_multiple_executors_and_pv_features

ca8a022

update log

b16cf80

Reenable Nextflow workflows on Kind CI

1b262ba

enable MV tests in Kind CI

286c1f2

paulineribeyre requested a review from nss10 July 24, 2026 15:56

paulineribeyre added the TestGen3WorkflowNextflow label Jul 24, 2026

remove comment

fb63fc5

paulineribeyre added 3 commits July 24, 2026 16:44

add test_pv_posix_ops

6a6720a

disable broken tests

9b78864

Merge branch 'master' of github.com:uc-cdis/gen3-code-vigil into enab…

dd54a09

…le-kind-nextflow

paulineribeyre removed the TestGen3WorkflowNextflow label Jul 27, 2026

paulineribeyre added 2 commits July 27, 2026 11:26

add missing script

5fb90cd

update comment

986831a

paulineribeyre changed the title ~~MIDRC-1292 Enable Nextflow tests in Kind CI~~ MIDRC-1292 Enable Nextflow tests in Kind CI + test_pv_posix_ops Jul 27, 2026

fix log

8aeb8da

paulineribeyre added the TestGen3Workflow label Jul 27, 2026

Merge branch 'master' into enable-kind-nextflow

ae38e39

paulineribeyre added 5 commits August 3, 2026 12:00

Merge branch 'master' of github.com:uc-cdis/gen3-code-vigil into enab…

ef26de0

…le-kind-nextflow

pull #581

b0b7571

merge the 2 PV tests

107a2e4

skip 10mb input

bfbdde4

auto task name

c34fa08

paulineribeyre mentioned this pull request Aug 3, 2026

TES tests for range param and PV issues #581

Closed

paulineribeyre changed the title ~~MIDRC-1292 Enable Nextflow tests in Kind CI + test_pv_posix_ops~~ MIDRC-1292 Enable Nextflow tests in Kind CI + test range param + test PV ops Aug 3, 2026

fix

f9f7151

paulineribeyre force-pushed the enable-kind-nextflow branch from 2c8c894 to f9f7151 Compare August 4, 2026 20:09

nss10 requested changes Aug 5, 2026

View reviewed changes

paulineribeyre added 3 commits August 6, 2026 13:48

range docstring

1f3daab

default log level to debug

6038ad1

update comment, rename script

92e287d

paulineribeyre requested a review from nss10 August 6, 2026 18:58

trigger CI

69c889e

rename script file ref

d3ceb5e

paulineribeyre added test_multiple_executors_and_pv_features and removed TestGen3Workflow labels Aug 10, 2026

		print(f"SKIP_BROKEN: {os.environ.get('SKIP_BROKEN')} => {SKIP_BROKEN}")


		# Test sequential file writes

		- Incremental/append writes to files already flushed to S3
		Note: unable to reproduce issue at this time

	# Test sequential file writes
	with open("output.txt", "w") as f:
	f.write("Initial sequential data\n")
	if not SKIP_BROKEN:
	with open("output.txt", "a") as f:
	f.write("Second sequential data\n")

Uh oh!

Conversation

paulineribeyre commented Jul 24, 2026

New Features

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jul 24, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jul 24, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Jul 24, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jul 24, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Jul 24, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jul 25, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Jul 27, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Jul 27, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jul 27, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Jul 27, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jul 30, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Jul 30, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Jul 30, 2026

Integration Tests

Uh oh!

github-actions Bot commented Aug 3, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Aug 3, 2026

Integration Tests

Uh oh!

github-actions Bot commented Aug 4, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Aug 4, 2026

Integration Tests

Uh oh!

nss10 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!