Skip to content

benchmarking.md

Latest commit

 

History

History
78 lines (56 loc) · 5.82 KB

benchmarking.md

File metadata and controls

78 lines (56 loc) · 5.82 KB
Raw

End-to-End Benchmarking

oar-bench (the unpublished oar-ocr-bench workspace crate) times complete OCR, structure, and VL page parsing on a fixed set of page images, and compares two runs. Run it from the repository root.

Run

Put the pages to measure in the gitignored benchmark-inputs/ directory, or pass image files and directories (top-level images only) with repeated --input flags. Then:

cargo run --release -p oar-ocr-bench --features cuda,nvml --bin oar-bench -- \
  run --output benchmark-results/base.json
  • --manifest selects the case list (default oar-ocr-bench/manifests/default.toml, one case per model architecture on cuda:0).
  • --case <NAME> (repeatable) runs a subset; --device auto|cpu|cuda:N|metal overrides every case's device.
  • Classic model names are auto-download registry names; VL checkpoints are local directories under models/<org>/<name>. Download them beforehand so loading time does not include network transfers.

Each case runs in a fresh subprocess, so memory peaks are per case and GPU memory is released between cases. The run prints a Markdown table and writes a JSON report; it exits nonzero if any case failed.

An explicitly requested accelerator that cannot be initialized fails the case instead of silently measuring a CPU fallback. auto records the device it actually selected.

Manifest

[inputs]
image_dirs = ["benchmark-inputs"]  # or images = ["page.png"]; optional max_pages

[defaults]
device = "cuda:0"
warmup = 2
repetitions = 5

[defaults.options]
cpu_threads = 4          # ORT intra-op threads and the Rayon pool
# batch_size = 1         # pages per pipeline call; the library picks its own image batch
# region_batch_size = 4  # classic region batch; external-layout VL and MinerU
# max_tokens = 4096      # VL generation budget
# gpu_memory_budget = 4294967296  # classic GPU tuning hint in bytes

[[cases]]
name = "ocr-tiny"
kind = "ocr"              # ocr | structure | vl
[cases.models]
detector = "pp-ocrv6_tiny_det.onnx"
recognizer = "pp-ocrv6_tiny_rec.onnx"
dictionary = "ppocrv6_tiny_dict.txt"

Cases may override device, warmup, repetitions, and [cases.options]. structure cases take a layout model (with layout_name such as PP-DocLayoutV3), optional OCR models, and optional table models. vl cases require model, the checkpoint's Hugging Face repo ID (for example PaddlePaddle/PaddleOCR-VL-1.5 or SoMarkAI/OXR-1.0), plus the local model_path; layout-composed models such as PaddleOCR-VL, GLM-OCR, TeleOCR, and OXR also need a PP-DocLayout layout_path. The ID never comes from the directory: most models are fine-tunes whose configs match their public base models, so the ID says what the directory holds.

Official accuracy evaluation

Add run --save-outputs predictions to write the last measured repeat's text to predictions/<case>/<image-stem>.md, outside inference timing. VL saves the text used for chars/s, OCR saves recognized text, and structure saves Markdown. Use a fresh directory per run: a non-empty case directory is rejected, as are duplicate image stems within a case.

  • OmniDocBench: use matching v1.5 annotations and images with the v1_5 evaluation branch. Each image needs a same-stem .md. Set the end2end config's ground_truth.data_path to the annotation JSON and prediction.data_path to predictions/<case>, then run the official python pdf_validation.py --config configs/end2end.yaml.
  • olmOCR-Bench: render PDF pages to images first, naming them <pdf-stem>_pg<1-based-page>_repeat1.png. The scorer requires bench_data/<method>/<pdf-relative-stem>_pg<page>_repeat1.md; arrange saved files into the PDF's relative category directories before running the official python -m olmocr.bench.benchmark --dir bench_data. Bench saves flat case directories and does not reconstruct those category paths or rename pages.

Measurements

Field Meaning
load_ms Device setup and pipeline construction
latency_ms Mean, p50, and p95 per page over all repetitions; a batched page takes the whole batch time
pages_per_second Measured pages divided by measured inference time
output_chars_per_second VL only; PageParser exposes no generated-token count
host_peak_bytes Linux VmHWM of the case process
gpu With nvml: device-wide used memory before loading and its sampled peak

Warmup runs are excluded. A VL page with parser diagnostics, such as an exhausted token budget or a failed region, fails the case rather than being timed. GPU memory is device-wide and sampled every 10 ms, so use an otherwise idle GPU and treat the peak as a lower bound. The sampled GPU honors CUDA_VISIBLE_DEVICES; on multi-GPU hosts also set CUDA_DEVICE_ORDER=PCI_BUS_ID so CUDA ordinals follow NVML's order.

Compare

cargo run --release -p oar-ocr-bench --bin oar-bench -- \
  compare benchmark-results/base.json benchmark-results/new.json --threshold 5%

For each case, latency and throughput (pages/s, and chars/s for VL) changes worse than the threshold are reported as regressions; memory peaks vary between identical runs and are shown as information only. Cases are matched by name, and inputs by page path, so keep the page images unchanged between the two runs. Cases whose configuration (other than device, warmup, and repetitions), pages, or actual devices differ are reported instead of compared; with nvml, the GPU model is part of the device, so the same cuda:0 on different GPUs is not compared. Without nvml, compare runs from the same machine. Environment differences (commit, CPU, features, build profile, OAR_* overrides) are printed as a note. The command exits nonzero on any regression, a case missing or failed in either report, or an incomparable case.