I have some "fishing pants" by Columbia that zip off to become swim-trunks. Nothing fancy, but they've held up well over several years of intermittent (vacation) use.

ETA: I doubt any convertible pant is going to pass as "business casual", but these aren't too bad if no one looks closely. Khaki color, no bulging cargo pockets, etc.