Hole-in-the-wall: Perception of 3D shape and affordances from static images in humans and machines


One popular toy for toddlers involves sorting block shapes into their respective holes. While toddlers require trial-and-error actions to sort blocks correctly, adults can rapidly see the appropriate solution through visual inspection alone. This feat requires an understanding of 3D shape and mental rotation. We study this task in a simplified vision-only setting by generating “shapes” of varying complexity using square matrices filled with connected binary regions, and “holes” by taking the negative region.“Fits” and “doesn’t fit” conditions are created while ensuring that shapes do not match exactly and that the total filled area is the same in both conditions. These matrices are rendered into black-and-white images (“bw”) and into more realistic rendered scenes. Human observers performed a single-interval fits/doesn’t fit task for two complexity levels for bw and rendered scenes. Performance was high for both bw (average …

Matthias Bethge
Matthias Bethge
Professor for Computational Neuroscience and Machine Learning & Director of the Tübingen AI Center

Matthias Bethge is Professor for Computational Neuroscience and Machine Learning at the University of Tübingen and director of the Tübingen AI Center, a joint center between Tübingen University and MPI for Intelligent Systems that is part of the German AI strategy.