ci: drop the deprecated tpchgen-cli flags, and cut the workflow's wall clock

tpchgen-cli 3.0.0 warns that `--format` goes away in 4.0.0 and that
`--parquet-compression` is deprecated, both in favour of a subcommand per
format. `tpchgen-cli parquet --compression=...` produces a byte-identical tree
at SF0.01, so this is a rename; `bench.sh` gets the same treatment for its
parquet, csv and sort-pushdown calls, and says which version it needs.

The step timings of the SF10 run say where the half hour goes, and it is not
the part that looks expensive:

    build base runner   990s   (835s cargo, 131s freeing disk space)
    build head runner   966s   (845s cargo,  96s freeing disk space)
    benchmark           461s   (410s measuring, 34s generating the data)

So generating the data is 34 seconds and needs nothing done to it, while the
builds are two minutes of deleting Android SDKs followed by fourteen minutes of
compiling dependencies that neither side changed. The builds now restore a
`Swatinem/rust-cache` entry, shared between the two sides because they are a
commit or two apart, written only by pushes to `main` so pull request runs do
not each save a near-identical copy; and the disk cleanup is skipped when the
runner already asked for `disk=large`, which is every run that gets the RunsOn
machine.

The warmup pass is now a read of the data files rather than a discarded pass of
the suite. Each round is a fresh process, so the page cache is the only thing a
warmup can carry between rounds, and reading the files fills it in seconds
where a pass of the suite cost as much as a measured round.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: Dandandan

.github/workflows/benchmark.yml

@@ -190,6 +190,11 @@ jobs:
         with:
           fetch-depth: 2
       - name: Free Disk Space (Ubuntu)
+        # Two minutes of deleting Android SDKs, which a release build of the
+        # workspace genuinely needs on `ubuntu-latest`'s 14GB. The RunsOn
+        # runner above asks for `disk=large` and has no such problem, so only
+        # the fallback pays for this.
+        if: vars.USE_RUNS_ON != 'true'
         uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be  # v1.3.1
       - name: Install Rust
         run: |
@@ -210,14 +215,28 @@ jobs:
           else
             git worktree add --detach "$BENCH_ROOT/$SIDE" HEAD
           fi
+      - name: Cache the dependency build
+        # A cold build of this is fourteen minutes, most of it dependencies
+        # that neither side changed. The action drops workspace crates from
+        # what it saves, so a hit rebuilds DataFusion and reuses the rest.
+        #
+        # `workspaces` because the build happens in a worktree outside the
+        # checkout, and one shared key for both sides because they are a commit
+        # or two apart and share a dependency graph. Only a push to `main`
+        # writes the cache: pull request runs would each save a near-identical
+        # copy of it.
+        uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6  # v2.9.2
+        with:
+          workspaces: ${{ env.BENCH_ROOT }}/${{ matrix.side }}
+          shared-key: benchmark-${{ env.CARGO_PROFILE }}
+          save-if: ${{ github.event_name == 'push' }}
       - name: Build benchmark_runner
         env:
           SIDE: ${{ matrix.side }}
-          CARGO_TARGET_DIR: ${{ runner.temp }}/target
         run: |
           cd "$BENCH_ROOT/$SIDE"
           cargo build --profile "$CARGO_PROFILE" -p datafusion-benchmarks --bin benchmark_runner
-          cp "$CARGO_TARGET_DIR/$CARGO_PROFILE/benchmark_runner" "$RUNNER_TEMP/benchmark_runner"
+          cp "target/$CARGO_PROFILE/benchmark_runner" "$RUNNER_TEMP/benchmark_runner"
       - name: Upload benchmark_runner
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a  # v7.0.1
         with:
@@ -283,12 +302,14 @@ jobs:
         # crate, but a ~4MB download instead of a build from source, since the
         # project attaches no binaries to its GitHub releases for a prebuilt
         # fetch to find. It also means this job needs no Rust toolchain at all.
+        #
+        # `parquet` as a subcommand rather than `--format parquet`: 3.0.0
+        # deprecated the flag form and warns that it goes away in 4.0.0.
         run: |
           mkdir -p "$DATA_DIR/tpch_sf$SCALE_FACTOR" "$RESULTS_DIR/base" "$RESULTS_DIR/head"
-          uv tool run --from 'tpchgen-cli==3.0.0' tpchgen-cli \
+          uv tool run --from 'tpchgen-cli==3.0.0' tpchgen-cli parquet \
             --scale-factor "$SCALE_FACTOR" \
-            --format parquet \
-            --parquet-compression 'ZSTD(1)' \
+            --compression 'ZSTD(1)' \
             --parts=1 \
             --output-dir "$DATA_DIR/tpch_sf$SCALE_FACTOR"
           du -sh "$DATA_DIR/tpch_sf$SCALE_FACTOR"
@@ -323,11 +344,13 @@ jobs:
               --path "$DATA_DIR" \
               --output "$output"
           }
-          # One discarded pass first, so no measured round is the only one
-          # paying to pull the parquet files into the page cache. It writes
-          # outside RESULTS_DIR, which is where compare.py looks for rounds.
-          echo "::group::warmup"
-          run_side base "$RUNNER_TEMP/warmup.json"
+          # Read the data once first, so no measured round is the only one
+          # paying to pull the parquet files into the page cache. Reading the
+          # files is all a warmup can carry between rounds -- each round is a
+          # fresh process -- and it takes seconds where a discarded pass of the
+          # suite took as long as a round.
+          echo "::group::warm the page cache"
+          find "$DATA_DIR" -type f -exec cat {} + > /dev/null
           echo "::endgroup::"
           for round in $(seq 1 "$ROUNDS"); do
             # Alternate which side goes first, so with an even ROUNDS

benchmarks/README.md

@@ -333,7 +333,10 @@ JSON plus the table are uploaded as the `tpch-comparison` artifact.
 
 The data is generated with the same `tpchgen-cli` settings `bench.sh data tpch`
 uses, from the tool's PyPI wheel rather than a source build, which keeps a Rust
-toolchain out of the benchmark job entirely.
+toolchain out of the benchmark job entirely. Generating SF10 takes well under a
+minute; what the workflow's half hour actually goes on is the two builds, so
+they restore a dependency cache and skip the `ubuntu-latest` disk cleanup when
+the runner has the disk already.
 
 Runners are shared machines, so treat the numbers as a signal rather than a
 measurement. A query has to clear three separate bars to fail the gate, and

benchmarks/bench.sh

@@ -668,7 +668,7 @@ data_tpch() {
     # check if tpchgen-cli is installed
     if ! command -v tpchgen-cli &> /dev/null
     then
-        echo "tpchgen-cli could not be found, please install it via 'cargo install tpchgen-cli'"
+        echo "tpchgen-cli could not be found, please install it via 'cargo install tpchgen-cli' (3.0 or newer)"
         exit 1
     fi
 
@@ -689,7 +689,7 @@ data_tpch() {
           echo " parquet files exist ($FILE exists)."
       else
           echo " creating parquet files using tpchgen-cli ..."
-          tpchgen-cli --scale-factor "${SCALE_FACTOR}" --format parquet --parquet-compression='ZSTD(1)' --parts=1 --output-dir "${TPCH_DIR}"
+          tpchgen-cli parquet --scale-factor "${SCALE_FACTOR}" --compression='ZSTD(1)' --parts=1 --output-dir "${TPCH_DIR}"
       fi
       return
     fi
@@ -701,7 +701,7 @@ data_tpch() {
           echo " csv files exist ($FILE exists)."
       else
           echo " creating csv files using tpchgen-cli binary ..."
-          tpchgen-cli --scale-factor "${SCALE_FACTOR}" --format csv --parts=1 --output-dir "${TPCH_DIR}/csv"
+          tpchgen-cli csv --scale-factor "${SCALE_FACTOR}" --parts=1 --output-dir "${TPCH_DIR}/csv"
       fi
       return
     fi
@@ -1308,7 +1308,7 @@ data_sort_pushdown() {
     TEMP_DIR="${DATA_DIR}/sort_pushdown_temp"
     mkdir -p "${TEMP_DIR}" "${SORT_PUSHDOWN_DIR}"
 
-    tpchgen-cli --scale-factor 1 --format parquet --parquet-compression='ZSTD(1)' --parts=3 --output-dir "${TEMP_DIR}"
+    tpchgen-cli parquet --scale-factor 1 --compression='ZSTD(1)' --parts=3 --output-dir "${TEMP_DIR}"
 
     # Rename: reverse alphabetical order vs key order
     mv "${TEMP_DIR}/lineitem/lineitem.3.parquet" "${SORT_PUSHDOWN_DIR}/a_part3.parquet"