The finale of Season 4 has an event that massively affects the show, and things seem to change from there. It's not an immediate plummet in quality, but it does definitely start to shift there.