As you say, under some conditions layers are laid down more quickly, and of different materials. Over time different materials also compress to different degrees. All of these conditions leave clues geologists can identify and interpret: "time passage of sequential" layers is still time passage, and additional information about context lets them interpret the scale of that time. (Similarly, tree rings grow at different rates under different conditions, but can be interpreted reliably.)