The colors were chosen for their ability to stand out from natural background lighting and for the general human ability to see them. (Red and Yellow are naturally easy for the human eye to see).

Finally a colorblind person wouldn't be impacted since the they can just count the lights. The top light is "stop" and the bottom is "go" regardless of color.