This is an example of an inverse problem in perception. The light that finally reaches our retina is the product of three things: the nature of the illuminant (what wavelengths are emitted), the surface reflectance properties of an object (what wavelengths are reflected), and the medium through which they travel (what wavelengths are filtered). Here is an illustration (panel A).

So basically there are three physical quantities that the visual system needs to recover from a single value (what actually arrives at your retina). That means that there are lots of possible combinations that can produce the exact same input to the visual system! If more light falls on our retina, how can we tell if it's because the light source got brighter, the object changed color or became more reflective, or if the air became less foggy?

Fortunately, there is a lot of information available in the world that helps us. For example, if there are several objects in a room, then we can see different surface reflectances under the same illumination (holding one of the variables constant). We also can make lots of assumptions based on our prior experiences, such as light usually comes from above etc.

When everything goes well, we achieve what is called luminance and color constancy that is -- we experience objects as having consistent surface reflectance properties despite changes in illumination. This is a very reasonable goal for our visual system: very few objects in the natural world quickly change color, but the illuminantion and medium (e.g. fog or mist) changes all the time every day.

These assumptions of our visual system can lead to fantastic illusions where constancy fails. Classic examples are when you see a sweater in a store under one illumination and then go outside and it's a completely different color. This is in fact the explanation for the dress illusion from a few years back -- depending on your assumption of the color of the light in the store (yellowish or whitish) the dress appears either blue and black or white and gold.

Here are a few other fun examples:

Because we take into consideration that shadows reduce the amount of light reaching our eye, we assume that surfaces in shadow are actually lighter than they appear. Here squares A and B are physically identical (exact same pixel values) but one appears much lighter than it is. Here is a color version and here is a version that shows the effect of the illuminant.

I find this one the most compelling -- the chess pieces are actually identical (not black and white); the only thing that is different is the pattern of the fog (here is the same image with no background/foreground).

Here is an example of the "light-from-above prior". This is actually the same image, just flipped 180 degrees. You can download it and rotate it yourself (or rotate your phone!) and you will see the one that sticks out become the one that is an indent and vice versa. If we assume that light is coming from above, if we have a convex object, the shadow would be below it; if it's concave, then the shadow would be at the top, below the rim/edge where light cannot reach. This is a 2D image so we don't have some of our other 3D cues to tell us about the shape of the surfaces here; however, because the image on the left has a shadow on top, we perceive it as concave/indented, while the image on the right, with the shadow on the bottom, is convex/ a bump. Flipping the image changes the position of the shadpw relative to the object and light source and so we perceive the shape differently. This is actually the same principle for how we would apply makeup to, for example, make our cheekbones stand out: you would put something light above the cheekbone and something dark below; this simulates the shadow that a pronounced cheekbone would cause and makes it appear more like a bump.

So what is the "true color" of an object? Color is not a physical property, but a psychological one / a property of the nature of our visual system and how it interacts with light. The physical property of objects that is relevant for this is surface reflectance, which we can describe objectively and independently of the visual system that detects the reflected light. If we had a different visual system, or, as shown above, if we just change the context or our assumptions about the world, objects can appear (i.e. we can experience them) different, but their properties are constant.