I gave up around season 4 too, but mostly because every episode felt like a bunch of unrelated moments at this point and I didn't have the patience to wait till the season finale where they would hopefully come together.

They had by then so many characters and subplots, a typical episode was like "Here are 10 minutes with Khaleesi, now five minutes with Aria, now 20 minutes with Tyrion, now a quick look at whoever that asshole is doing, followed by a scene with some guy who wasn't there since 2 seasons ago, now a few more minutes with Jon Snow or whoever and roll the credits."

It just made me feel completely unattached to anything and anybody on this show.