You could store the audio clips as a Numpy memory map, and sample from it. This way, you will be able to sample clips without loading everything into memory. One library you can use is mmap-ninja.

Disclaimer: I'm the author of mmap-ninja.