Reproducible tasks
Clear task definitions, scoring rules, evaluation conditions, and dataset documentation.
Open evaluation Coming soon
A benchmark for
understanding buildings.
We’re developing an evaluation suite for material recognition, defect detection, and building-focused visual reasoning. Compare how models perform on the details that matter in the physical world.
Get the benchmark release
Get the first release, including task definitions, evaluation methods, and model comparisons.
Planned evaluation areas
Five areas
of visual understanding.
Our evaluation work spans recognizing individual features, locating defects, reasoning about a scene, and transferring learning from synthetic environments to real imagery.
| Evaluation area | What it asks |
|---|---|
| Material recognition | Classification across exterior material categories and viewing conditions. |
| Feature localization | Finding and delineating meaningful building features in imagery. |
| Visible anomaly detection | Identifying regions that match a defined visual defect or anomaly. |
| Building-focused visual reasoning | Answering structured questions about visible building evidence. |
| Synthetic-to-real transfer | Measuring performance on real-world data after synthetic-data training. |
What a useful release includes
Tasks, methods, and results.
Comparable baselines
Relevant models and methods evaluated under a common protocol, with model versions identified.
Detailed results
Results broken down by task, material, scene conditions, and image quality, with examples that show where models struggle.