From the point of a photographer trying to recreate the colors an object as it was seen in the lighting that was there the grey card works just like you say. I think we actually agree like you say, but just the see the problem from different perspectives. I work as a theatrical lighting designer so my go to light source in my mind is anything but a source with a broad and continous spectrum. Anyways, colours are wierd and fantastic and this conversation really made me think this through a lot so thanks a lot for that :)