<p>Orientation processing is central to visual perception, yet the contributions of different image features in natural scene orientation processing remain unresolved. Here, we present converging behavioral and neural evidence that human orientation judgments in real-world scenes depend more strongly on contours delineating object structure than on orientation energy from filter-based computations. In Study 1, participants judged the average orientation of image patches chosen to maximize the discrepancy between contour-based and filter-based orientation estimates. Judgments consistently matched contour-based estimates, indicating a perceptual bias for object boundaries. In Study 2, we modeled fMRI responses from the Natural Scenes Dataset using three image-computable models: Photo–Steerable Pyramid, Line drawing–Steerable Pyramid, and a Contour-based approach. Models prioritizing contour structure better explained the neural data and identified discrete patterns of orientation preferences. Collectively, these findings show that the human visual system encodes scene orientation by extracting contours, the fundamental components of shapes, over filter-based orientation energy. This challenges prevailing models of visual representation and underscores the importance of boundary information in natural vision.</p>