CARScenes Release Workspace

This repository contains the finalized CARScenes annotation workspace, release artifacts, benchmark tooling, and NeurIPS paper sources.

Public Artifact URLs

• GitHub code and release workspace: https://github.com/Croquembouche/CARScenes
• Hugging Face dataset landing page: https://huggingface.co/datasets/williamhe0712/CARScenes

What Is Here

• dataset/: unified final label JSON files and source image layout.
• release/carscenes-v1/: canonical annotation-only release, schema, splits, reports, and Croissant metadata.
• configs/hf_multimodal_model_presets.json: public-weight model presets for the local open-model benchmark workflow.
• scripts/train_hf_multimodal_qlora.py: generic QLoRA trainer for public Hugging Face multimodal models.
• scripts/run_hf_multimodal_benchmark.py: zero-shot or adapter-backed benchmark runner that emits CARScenes-compatible prediction JSONL files.
• scripts/evaluate_carscenes_predictions.py: scores model predictions against a CARScenes split.
• scripts/generate_benchmark_report.py: aggregates multiple runs into CSV, Markdown, and LaTeX benchmark tables.
• scripts/prepare_alpamayo_side_experiment.py: exports the 25-image Alpamayo side-experiment bundle.
• scripts/build_carscenes_release.py: rebuilds the release from the unified labels.
• scripts/validate_carscenes_release.py: validates the release records and split manifests.
• scripts/carscenes_review_dashboard.py: local web app for benchmark upload/evaluation and second-pass human review.
• paper/: NeurIPS manuscript source and final output PDF.

Benchmark Splits

• release/carscenes-v1/splits/dev-500.jsonl: fixed 500-example benchmark split used for the main local model table. In the paper this is referred to as Benchmark-500.
• release/carscenes-v1/splits/gold-test-100.jsonl: smaller curated gold audit split.
• release/carscenes-v1/splits/gold-agreement-25.jsonl: second-pass human agreement subset.

Core Commands

Rebuild and validate the canonical release:

python scripts/build_carscenes_release.py
python scripts/validate_carscenes_release.py

Evaluate one prediction JSONL against the main benchmark split:

python scripts/evaluate_carscenes_predictions.py \
  --reference release/carscenes-v1/splits/dev-500.jsonl \
  --predictions path/to/predictions.jsonl

Generate aggregate benchmark tables from multiple model runs:

python scripts/generate_benchmark_report.py \
  --manifest templates/benchmark_runs_template.csv \
  --output-dir runs/benchmark/table5_main_finetuned_report

Launch the unified local review dashboard:

python scripts/carscenes_review_dashboard.py

Then open http://127.0.0.1:8770/.

Open-Model Benchmark Workflow

The current paper deliberately presents the local benchmark as a split-pure reference study rather than as a broad leaderboard.

Paper-facing grouped benchmark table

• Qwen2-VL-2B zero-shot: compact no-schema same-family reference on Benchmark-500.
• Qwen2-VL-2B LoRA split-pure: clean adapter trained only on the frozen Train 4,592 split and evaluated on Benchmark-500.
• InternVL2-1B LoRA legacy control: older smaller public-weight control retained for failure-profile context, not a split-pure comparison row.

Paper-facing zero-shot control

• InternVL2-1B: weaker public-weight zero-shot control used to show the off-the-shelf gap.

Experimental feasibility paths not promoted into the paper

• LLaVA-OneVision-Qwen2-0.5B-OV: parseable under the compact/no-schema/json-only prompt recipe, but still collapsed on scene, directionality, and severity smoke gates.
• LLaVA-OneVision-Qwen2-7B-OV: now supported by the local launcher and sharded eval wrapper, but the bounded smoke path was not promotable on this hardware. Two-GPU QLoRA probes OOM'd; a capped three-GPU smoke run completed, but every logged step had loss=0.0, grad_norm=0.0, and eval_loss=nan, so the model was not advanced into the paper tables.

Manifest templates

• Main paper table: templates/benchmark_runs_template.csv
• Zero-shot companion: templates/benchmark_runs_zero_shot_template.csv

Launcher script

Use scripts/run_table5_open_models.sh for the current paper rows or for experimental feasibility reruns:

scripts/run_table5_open_models.sh qwen2_eval
scripts/run_table5_open_models.sh internvl_zero_shot
scripts/run_table5_open_models.sh internvl_eval
scripts/run_table5_open_models.sh report_main
scripts/run_table5_open_models.sh report_zero_shot
scripts/run_table5_open_models.sh llava05_zero_shot
scripts/run_table5_open_models.sh llava05_train
scripts/run_table5_open_models.sh llava05_eval
scripts/run_table5_open_models.sh llava7_zero_shot
scripts/run_table5_open_models.sh llava7_train
scripts/run_table5_open_models.sh llava7_eval
scripts/run_table5_open_models.sh alpamayo_prep

Local Training and Inference

Current paper evidence

Regenerate the control rows and paper-facing reports with:

scripts/run_table5_open_models.sh qwen2_eval
scripts/run_table5_open_models.sh internvl_zero_shot
scripts/run_table5_open_models.sh internvl_eval
scripts/run_table5_open_models.sh report_main
scripts/run_table5_open_models.sh report_zero_shot

Experimental LLaVA feasibility path

The compact/no-schema/json-only materialized datasets used for the LLaVA feasibility probes are:

• runs/cache/train_qwen_compact_jsononly_noschema.jsonl
• runs/cache/val_dataset_500_compact_jsononly_noschema.jsonl

The generic HF trainer and benchmark runner remain available for bounded feasibility checks:

scripts/run_table5_open_models.sh llava05_zero_shot
scripts/run_table5_open_models.sh llava05_train
scripts/run_table5_open_models.sh llava05_eval
scripts/run_table5_open_models.sh llava7_zero_shot
scripts/run_table5_open_models.sh llava7_train
scripts/run_table5_open_models.sh llava7_eval

Alpamayo Side Experiment

Prepare a local bundle for the 25-image subset:

python scripts/prepare_alpamayo_side_experiment.py \
  --selection-jsonl audit_outputs/benchmark500_random25_20260325/selected_records.jsonl \
  --output-dir runs/alpamayo/benchmark500_random25_20260325 \
  --copy-images

This writes:

• requests.jsonl: one request record per sampled image with prompts and resolved image path
• prompt.txt: system and user prompt text
• README.md: run instructions for manual or external Alpamayo execution
• images/: copied images when --copy-images is enabled

Notes

• runs/ and audit_outputs/ are gitignored by default.
• The Hugging Face benchmark tooling can use either the historical Qwen prompt family or the compact prompt configs, and it normalizes parsed outputs into the released JSON schema before scoring.
• Qwen2-VL-2B LoRA split-pure is the current paper-facing adapted reference; the historical Qwen checkpoint is archived outside the main table because it predates the final split freeze.
• The current paper-facing benchmark uses the clean Qwen split-pure row plus the matching zero-shot reference; the LLaVA feasibility paths are kept in the repo as experimental infrastructure, not as promoted paper rows.

Licensing

• CARScenes annotations, schema, split manifests, and metadata: CC BY 4.0
• Repository code: MIT
• Source images remain under the original upstream dataset licenses and are not relicensed here.