|
| 1 | +--- |
| 2 | +name: add-benchmarks |
| 3 | +description: Guidelines for designing or extending DataFusion benchmarks. |
| 4 | +--- |
| 5 | + |
| 6 | +# Benchmark Design |
| 7 | + |
| 8 | +Follow these guidelines when designing or adding a new benchmark. |
| 9 | + |
| 10 | +## Design principles |
| 11 | + |
| 12 | +### Use the highest-level interface |
| 13 | + |
| 14 | +Use the highest-level interface possible, preferably SQL, while keeping the work |
| 15 | +around the operator being measured as cheap as possible. When benchmarking a |
| 16 | +function, exercise its evaluation path instead of benchmarking internal utility |
| 17 | +functions in isolation. |
| 18 | + |
| 19 | +This makes benchmarks easier to maintain and helps assess how much an optimization |
| 20 | +matters to end-to-end runtime. It also helps avoid spending time optimizing code |
| 21 | +that accounts for only a small fraction of the total runtime. |
| 22 | + |
| 23 | +Practical criteria for choosing SQL or Rust benchmarks: Try implementing the benchmark in SQL first. If a Rust microbenchmark still seems like a better fit, use Criterion. |
| 24 | + |
| 25 | +### Vary the key workload axes |
| 26 | + |
| 27 | +First identify the key axes to vary. For example, for a join benchmark: |
| 28 | + |
| 29 | +- Input size on each side. |
| 30 | +- Join-filter selectivity. |
| 31 | +- ... |
| 32 | + |
| 33 | +Then choose benchmark cases that exercise representative variations. Full |
| 34 | +combinatorial coverage is unnecessary; focus on typical workloads that reflect |
| 35 | +real use cases. |
| 36 | + |
| 37 | +When adding benchmark queries, simply tag each query with the decision made |
| 38 | +for every axis. For example, for a join benchmark with input sizes/filter selectivity |
| 39 | +to tune: |
| 40 | + |
| 41 | +```sql |
| 42 | +-- Q1: Small left input, large right input; 0.1% of pairs match. |
| 43 | +SELECT * |
| 44 | +FROM generate_series(1, 100) AS l |
| 45 | +JOIN generate_series(1, 100000) AS r |
| 46 | + ON (l.value + r.value) % 1000 = 0; |
| 47 | + |
| 48 | +-- Q2: Medium inputs on both sides; no filter. |
| 49 | +SELECT * |
| 50 | +FROM generate_series(1, 1000) AS l |
| 51 | +CROSS JOIN generate_series(1, 1000) AS r; |
| 52 | +``` |
| 53 | + |
| 54 | +## SQL benchmarks |
| 55 | + |
| 56 | +For implementation details, see the |
| 57 | +[SQL benchmark README](../../../benchmarks/sql_benchmarks/README.md). |
| 58 | + |
| 59 | +1. **Keep other operators cheap.** |
| 60 | + |
| 61 | + When a SQL benchmark targets a specific operator, keep the work done by |
| 62 | + other operators as lightweight as possible. For example, use a data source |
| 63 | + such as `generate_series()` instead of a Parquet scan so scan overhead does |
| 64 | + not dominate the measurement. See the [`nlj` benchmark](../../../benchmarks/sql_benchmarks/nlj/) for examples. |
| 65 | + |
| 66 | +2. **Integrate with the top-level benchmark script.** |
| 67 | + |
| 68 | + Ensure the benchmark can be prepared and run through `bench.sh`: |
| 69 | + |
| 70 | + ```bash |
| 71 | + # Run from the benchmarks directory. |
| 72 | + |
| 73 | + # Generate any required dataset. |
| 74 | + ./bench.sh data new_bench |
| 75 | + |
| 76 | + # Run the benchmark. |
| 77 | + ./bench.sh run new_bench |
| 78 | + ``` |
| 79 | + |
| 80 | +3. **Keep query runtimes practical.** |
| 81 | + |
| 82 | + Tune the workload so each query takes roughly a few seconds per execution. |
| 83 | + This helps reduce the relative impact of timing noise while keeping the |
| 84 | + suite practical to run. |
0 commit comments