If I understand correctly, you have a combination wall switch that controls the light separately from the fan instead of 2 separate switches (one for the light and one for the fan). If this is the case, you can install wire nuts on the second set of black and white wires and leave them in the electrical box where you are installing the new light.

You probably should go the added step and disconnect the second set of wires at the switch as well, and put a wire nut on this end too. It is for added safety in the off chance someone flips the "fan" switch on and leaves it on. It would at least not have power running through the wire to nothing.

Most people don't bother with wire nutting the abandoned wires at both ends, but it is the right thing to do and a good practice to be in. You dont have to remove the extra wires for any reason.