I would prime it. Add filler where needed. Lightly sand filler making sure not to sand through the primer. Wearing a respirator while doing so. then paint. And not be to fussy with the finish.

To get a perfect surface without filler added, youd have to sand the old paint smooth. Scraping it will leave it very uneven and painting over it will make it look ugly. Up to you though.