Abstract – How we perceive property of surfaces with distinct geometry and reflectance under various illumination conditions is not fully understood. One widely studied approach to understanding perceptual surface property is to derive statistics from images of surfaces with the goal of constructing models that can estimate surface property attributes. This paper presents machine-learning-based methods to estimate the lightness and glossiness of surfaces. Instead of deriving image statistics and building estimation models on top of them, we use deep networks to estimate the perceptual surface property directly from surface images. We adopt the attention models in our networks to allow the networks to estimate the surface property based on features in certain parts of images. This approach can rule out image variations due to geometry, reflectance, and illumination when making the estimations. The networks are trained with perceptual lightness and glossiness data obtained from psychophysical experiments. The trained deep networks provide accurate estimations of the surface property that correlate well with human perception. The network performances are compared with various image statistics derived for the estimation of perceptual surface property.