That’s exactly the difference between a colon and a semicolon.

A colon explicitly relates the two statements. In your example, a colon would indicate that he spent three hours in the library because he couldn’t find the book. A semicolon would leave it open to interpretation: he couldn’t find the book, but that’s not necessarily why he spent three hours in the library.

So yes you can use them in the same sentence but not interchangeably. They have different meanings