If they don't turn off, and you haven't changed the wiring from a known working configuration, then change the switch. Even if the light strings are non-dimming, they should at least turn off.

If you have changed the wiring configuration since it was last known to work, consult the switches manual. There were a few configurations possible to accommodate different lighting types. Also check that neutral isn't miswired somewhere and actually hot.