It really depends on making (and training) an adaptation to your per-frame detector to incorporate prediction priors, then devising a method for making the prediction priors from previous frames' output. I'm not that familiar with the specifics of your particular problem to add much more, I'm afraid.