Psychophysical studies suggest that different colors have different perceptual status: red and blue for example are thought of as elementary sensations whereas yellowish green is not. The dominant account for such perceptual asymmetries attributes them to specificities of the neuronal representation of colors. Alternative accounts involve cultural or linguistic arguments. What these accounts have in common is the idea that there are no asymmetries in the physics of light and surfaces that could underlie the perceptual structure of colors, and this is why neuronal or cultural processes must be invoked as the essential underlying mechanisms that structure color perception. Here, we suggest a biological approach for surface reflection properties that takes into account only the information about light that is accessible to an organism given the photopigments it possesses, and we show that now asymmetries appear in the behavior of surfaces with respect to light. These asymmetries provide a classification of surface properties that turns out to be identical to the one observed in linguistic color categorization across numerous cultures, as pinned down by cross cultural studies. Further, we show that data from psychophysical studies about unique hues and hue cancellation are consistent with the viewpoint that stimuli reported by observers as special are those associated with this singularity-based categorization of surfaces under a standard illuminant. The approach predicts that unique blue and unique yellow should be aligned in chromatic space while unique red and unique green should not, a fact usually conjectured to result from nonlinearities in chromatic pathways.