SCOUT: A Utility Based Agent for Open-World Interactive Object Search

Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search
Imen Mahdi¹ · Matteo Cassinelli² · Fabien Despinoy² · Tim Welschehold¹ · Abhinav Valada^1
1 University of Freiburg    ² Toyota Motor Europe

---

SCOUT is a method that searches directly over 3D scene graphs using learned relational utility scores (room-object containment, object-object co-occurrence)

Overview

Open-world interactive object search in household environments requires understanding semantic relationships between objects and their surrounding context to guide exploration efficiently. This repository provides SCOUT — our proposed method that searches directly over 3D scene graphs using learned relational utility scores (room-object containment, object-object co-occurrence). This reposotory provides examples of parts of the pipeline described in our paper including: the dataset generation pipeline, training utility scoring functions, and defining utility based search agents.

---

Installation

We recommend using uv for installation. This repository requires the evaluation benchmark SymSearch, presented in our paper.

git clone https://github.com/rl-uni-freiburg/scout.git
cd scout
uv sync
source .venv/bin/activate

To evaluate our agents on Symsearch, you need to download the InteriorGS dataset.

To set up the dataset:

0. Install Huggingface cli if you do not have it installed yet.
1. Log in to Huggingface: hf auth login
2. Accept the terms of service and request access to the dataset at spatialverse/InteriorGS
3. Run the setup script, which downloads the scenes and adds the SymSearch annotations:

python -m symsearch.download_datasets

Scout Utility Estimators

Scout provides learned and cosine-similarity-based estimators for scoring containement probabilities between queries and room categories or cooccurence probabilities between queries and objects.

---

Loading Pretrained Models and Scoring Queries

import pandas as pd
from scout.utility_estimators import LearnedUtilityEstimator

def display_scores(estimator, queries, targets):
    scores = estimator.compute_relevance_scores(queries, targets)
    df = pd.DataFrame(scores, index=queries, columns=targets)
    df.index.name   = "query"
    return df.style.background_gradient(axis=None, cmap="RdYlGn").format("{:.2f}")

queries = ["plate", "book", "toothbrush", "pillow", "blanket"]

# Scores how likely a query relates to a given room category
contain_estimator = LearnedUtilityEstimator.load("checkpoints/contain_sbert.ckpt")
rooms   = ["kitchen", "bedroom", "living room", "office", "bathroom"]
display_scores(contain_estimator, queries, rooms)


# Scores how likely a query co-occurs with a given object
cooccur_estimator = LearnedUtilityEstimator.load("checkpoints/cooccur_sbert.ckpt")
objects = ["sofa", "desk", "lamp", "bed", "sink", "table"]
display_scores(cooccur_estimator, queries, objects)

---

Optional: Cosine Similarity with Different Embedders

Instead of a learned estimator, you can score queries using cosine similarity in embedding space directly. This requires no pretrained checkpoint.

from scout.registry import EMBEDDERS_REGISTRY, load_embedder
from scout.utility_estimators import CosineSimilarityEstimator

print(EMBEDDERS_REGISTRY.keys())
# ['clip', 'sbert', 'openai']

# Swap "clip", "sbert", or "openai" to try different embedders
embedder = load_embedder("clip")
similarity_estimator = CosineSimilarityEstimator(embedder)

display_scores(similarity_estimator, queries, rooms)
display_scores(similarity_estimator, queries, objects)

---

Training Your Own Model

Scout includes a training script that reads a YAML config file. You can easily try different model sizes or embedders. Note: embedder_name should be one of the registered embedders. You can check embdeers.py to learn how to add your own.

Config Format

Configs live in scout/train/configs/<relation>.yaml. Example, you can change the embedder to clip in (contain.yaml):

embedder_name: clip # (`clip`, `sbert`, `openai`)  
layers: [512, 256, 128, 1]
binary: false # `true` → BCE loss + sigmoid output; `false` → MSE loss
pretrain_epochs: 5
epochs: 10
batch_size: 256
data_path: "data/contain.csv" # Path to the CSV dataset directory

Running Training

python -m scout.train contain      # trains with contain.yaml config
python -m scout.train cooccur      # trains with cooccur.yaml config

The checkpoint is saved to checkpoints/<relation>_<embedder_name>.ckpt, e.g. checkpoints/contain_clip.ckpt.

The resulting checkpoint can be loaded with:

cooccur_estimator = LearnedUtilityEstimator.load("checkpoints/contain_clip.ckpt")
display_scores(cooccur_estimator, queries, objects)

Generate your own data to train on

We provide an example pipeline in generate_data.ipynb to generate the csv files.

Evaluating SCOUT

Simplest way to do so is:

python -m  scripts.evaluate --agents SCOUT

or

python -m  scripts.evaluate \
    --agents SCOUT \
    --seeds 0 1 2 3 4\
    --benchmark InteriorGS \
    --n-tasks 200 \
    --max-steps 60 \
    --log-dir ./logs

Comparing SCOUT vs. SCOUT_CLIP ablation

To compare the default SCOUT (SBERT checkpoints) against the CLIP-trained variant, pass both agent names:

python scripts/evaluate.py --agents SCOUT SCOUT_CLIP

SCOUT_CLIP is pre-registered in scout/utility_agent.py and loads checkpoints/cooccur_clip.ckpt and checkpoints/contain_clip.ckpt — the models you trained with the CLIP embedder in the training step above.

Note: Make sure the CLIP checkpoints exist before running. Train them first by setting embedder_name: clip in both scout/train/configs/contain.yaml and scout/train/configs/cooccur.yaml as described above. Checkpoints will be saved to checkpoints/contain_clip.ckpt and checkpoints/cooccur_clip.ckpt.

Aggregating and Comparing Results

Once evaluation logs are written to ./logs, aggregate and compare results capped at 50 steps:

python scripts/aggregate_results.py --max-steps 50

Citation

If you use SCOUT in your research, please cite our paper:

@misc{mahdi2026relationalsemanticreasoning3d,
  title     = {Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search},
  author    = {Imen Mahdi and Matteo Cassinelli and Fabien Despinoy and Tim Welschehold and Abhinav Valada},
  year      = {2026},
  eprint    = {2603.05642},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url       = {https://arxiv.org/abs/2603.05642},
}

and the InteriorGS dataset used in evaluation:

@misc{InteriorGS2025,
  title        = {InteriorGS: A 3D Gaussian Splatting Dataset of Semantically Labeled Indoor Scenes},
  author       = {SpatialVerse Research Team, Manycore Tech Inc.},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/datasets/spatialverse/InteriorGS}}
}

---

License

SCOUT is released under CC BY-NC-SA 4.0.

The underlying InteriorGS dataset is subject to its own Terms of Use — please review and accept them before downloading the data.

---

Contact

For questions about the code, please open an issue or reach out directly:

Imen Mahdi — mahdi@cs.uni-freiburg.de — rl.uni-freiburg.de/people/mahdi

Abhinav Valada — valada@cs.uni-freiburg.de — rl.uni-freiburg.de/people/valada