[FEATURE] Add T2I‑CompBench++ Metric

[minettekaum]

✨ What You’ll Do

Adding more metrics makes Pruna’s evaluations even sharper — so we’re thrilled to welcome T2I-CompBench++ (paper, project)! 🎉

With 8 000 prompts spanning attributes, relationships, numeracy, and complex compositions — plus fresh detection-based metrics and multimodal LLM analyses — this benchmark is a huge step forward for testing compositionality, spatial reasoning, and numeracy in generative models.

---

📦 Data & Dependency

• Download the T2I-CompBench(++) dataset and include it in our evaluation assets.
• Install dependencies from the benchmark’s evaluation toolkit (BLIP-VQA, UniDet, 3-in-1 metrics).
• Ensure detection models for 3D spatial relationships and numeracy are available.

---

📐 Metric Class

For the full deep-dive on Pruna metrics (class structure, state handling, tests, registration), check out our guide: Customize a Metric.

Quick Overview

1. File & Class – Create pruna/evaluation/metrics/metric_t2i_compbench.py (or src/pruna/... if your repo is under src/) and subclass StatefulMetric.
2. Configuration – Pick a sensible call_type (likely per_batch for efficiency) and set metric_name = "t2i_compbench".
3. State Handling – Define and register any internal state via self.add_state(...), e.g.:

   self.add_state("scores", default=[], dist_reduce_fx="cat")

4. Core Methods - update(x, gt, outputs) → Feed generated images + prompts into the benchmark’s evaluation routines (BLIP-VQA for attribute binding, UniDet for spatial reasoning, 3-in-1 metric for complex prompts). Accumulate scores.
   compute() → Aggregate category scores and return a result dictionary.
5. Registration – Use @MetricRegistry.register("t2i_compbench") so it’s callable by name.
6. Testing & Docs – Write unit tests (tests/evaluation/metrics/test_t2i_compbench.py) and update user_manual/evaluation.rst.

---

✅ Acceptance Criteria

• Efficient Loading → Dataset + models (detection nets, MLLMs) are loaded once in the constructor.
• Correct Output → compute() returns meaningful scores capturing compositionality, 3D spatial reasoning, and numeracy (per the paper).
• Tests & Docs Updated → All tests pass; docs explain how to use the metric.

---

❓ Questions?

Hop into our Discord channel if you get stuck. We’re excited to see this comprehensive benchmark join Pruna’s evaluation lineup! 🚀

[pranayyb]

Hey @minettekaum I would like to work on this

[davidberenstein1957]

Hi @pranayyb, that is great. Do you have any idea where to start? As mentioned above, we've got some great examples of how to customize a metric but also have e more general guide https://docs.pruna.ai/en/stable/docs_pruna/contributions/how_to_contribute.html

[pranayyb]

I have gone through the todos but it would be great if you guide on how should i approach for this. I do have 2 PRs merged in pruna so i do have idea about the codebase so i am familiar with it.
Thanks :)

[pranayyb]

hey @davidberenstein1957 , I am not able to understand what you mean by adding dataset and dependencies as it exists as a repo and not as a package on pypi, could you please tell what I should do?

[davidberenstein1957]

hey @davidberenstein1957 , I am not able to understand what you mean by adding dataset and dependencies as it exists as a repo and not as a package on pypi, could you please tell what I should do?

Hi @pranayyb, A benchmark often consists of a dataset and evaluation metrics. The T2I-CompBench++ dataset was evaluated using a set of metrics. This specific example aims to add to the T2I-CompBench++ metrics, not the dataset itself. The metrics themselves rely on dependencies, which need to be added along the pruna implementation for those metrics. Does that clarify your question?