Skip to content

[Bug] Concurrent parse_junit --run-id uploads to one run drop results and still exit 0 #438

Description

@marksaxer-cbs

Description

trcli ... parse_junit --run-id <RUN> does not appear to be safe to run concurrently against the same run. When several CI shards each upload their own JUnit report to one shared run (created with include_all=false), results are silently dropped and the CLI still exits 0.

Two observable failure modes:

  • Loud: a shard fails with
    Field :results contains one or more invalid results (case C<id> unknown or not part of the test run),
    for cases that were in its own report.
  • Silent: in larger fan-outs, a whole module's results disappear from the run even though that shard's upload step logged success and exited 0.

Which shards survive is non-deterministic, depending on upload timing.

Steps to reproduce

  1. Create a run with include_all=false and an empty/partial case set.
  2. Prepare 4 JUnit reports covering different cases (e.g. 116 + 43 + 14 + 17 = 190 cases).
  3. Fire 4 uploads concurrently, all targeting the same run:
    trcli -y -h https://<instance>.testrail.io/ --project <P> -u <user> -k <key> \
      parse_junit --suite-id <S> --run-id <RUN> \
      --case-matcher auto --update-existing-cases yes -f report-<n>.xml
    
  4. Compare the run's final case count and per-module presence against the inputs.

Expected result

All 190 results land in the run regardless of upload timing/concurrency. Concurrent uploads to a shared run should be safe, or the tool should fail loudly rather than lose data and still exit 0.

Actual result

Only some cases land (observed: 159 of 190; 31 dropped). Losers either error with case C<id> not part of the test run, or — in larger fan-outs — disappear silently while the upload step reports success and exits 0.

Severity / Reproducibility

  • Severity: S2 Major — silent data loss in reporting; CI result aggregation blocked, no in-tool workaround.
  • Reproducibility: Intermittent (timing-dependent) — reliably reproducible on demand with as few as 4 concurrent uploads.

Versions / environment

  • trcli: 1.14.3
  • Python: 3.11
  • pytest: 7.x; JUnit XML via pytest
  • CI provider: GitHub Actions
  • OS: Ubuntu 22.04 x86_64

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions