Description
trcli ... parse_junit --run-id <RUN> does not appear to be safe to run concurrently against the same run. When several CI shards each upload their own JUnit report to one shared run (created with include_all=false), results are silently dropped and the CLI still exits 0.
Two observable failure modes:
- Loud: a shard fails with
Field :results contains one or more invalid results (case C<id> unknown or not part of the test run),
for cases that were in its own report.
- Silent: in larger fan-outs, a whole module's results disappear from the run even though that shard's upload step logged success and exited
0.
Which shards survive is non-deterministic, depending on upload timing.
Steps to reproduce
- Create a run with
include_all=false and an empty/partial case set.
- Prepare 4 JUnit reports covering different cases (e.g. 116 + 43 + 14 + 17 = 190 cases).
- Fire 4 uploads concurrently, all targeting the same run:
trcli -y -h https://<instance>.testrail.io/ --project <P> -u <user> -k <key> \
parse_junit --suite-id <S> --run-id <RUN> \
--case-matcher auto --update-existing-cases yes -f report-<n>.xml
- Compare the run's final case count and per-module presence against the inputs.
Expected result
All 190 results land in the run regardless of upload timing/concurrency. Concurrent uploads to a shared run should be safe, or the tool should fail loudly rather than lose data and still exit 0.
Actual result
Only some cases land (observed: 159 of 190; 31 dropped). Losers either error with case C<id> not part of the test run, or — in larger fan-outs — disappear silently while the upload step reports success and exits 0.
Severity / Reproducibility
- Severity: S2 Major — silent data loss in reporting; CI result aggregation blocked, no in-tool workaround.
- Reproducibility: Intermittent (timing-dependent) — reliably reproducible on demand with as few as 4 concurrent uploads.
Versions / environment
- trcli: 1.14.3
- Python: 3.11
- pytest: 7.x; JUnit XML via pytest
- CI provider: GitHub Actions
- OS: Ubuntu 22.04 x86_64
Description
trcli ... parse_junit --run-id <RUN>does not appear to be safe to run concurrently against the same run. When several CI shards each upload their own JUnit report to one shared run (created withinclude_all=false), results are silently dropped and the CLI still exits0.Two observable failure modes:
Field :results contains one or more invalid results (case C<id> unknown or not part of the test run),for cases that were in its own report.
0.Which shards survive is non-deterministic, depending on upload timing.
Steps to reproduce
include_all=falseand an empty/partial case set.Expected result
All 190 results land in the run regardless of upload timing/concurrency. Concurrent uploads to a shared run should be safe, or the tool should fail loudly rather than lose data and still exit
0.Actual result
Only some cases land (observed: 159 of 190; 31 dropped). Losers either error with
case C<id> not part of the test run, or — in larger fan-outs — disappear silently while the upload step reports success and exits0.Severity / Reproducibility
Versions / environment