Seems reasonable to quit then, but as others have said it doesn't seem very healthy, and you probably should look into why this happens.

The point of most books is also not plot twists, and not solving a puzzle. Have you tried books with very little plot happening, like "In search of the time lost", by Proust? What about literature from times/places with different storytelling tradition, like medieval epics, ancient Greek poems? How do you feel about classics, that everyone more or less knows the plot of? Or would you be able to enjoy a novel if you deliberately spoiled it to yourself first?