Yes, the height difference will make a difference. Definitely fix that before you lay the finished floor or you'll feel it every time you step on it. But it's easy to fix. Just use some strips of heavy paper or thin cardboard on top of the joists and then screw down the OSB on top of them. As many layers as necessary to make up the difference.