Had a similar problem. Wall had a light orange peel texture and after spackeling it looked horrible with all the smooth spots sticking out like a sore thumb. Had to skim it to fix it. TLDR just go straight to skimming.