Back in the day the key was weights. If two things combine to give a new substance that is heavier (say iron + oxygen to iron oxide) then it is obviously a compound. If a substance cleanly decomposes to two new substances, then the new substances are simpler and might be elements (say hydrogen peroxide to oxygen and water). After building up lots of these reaction it became clear that some things (e.g carbon) couldnt be decomposed into anything simpler and must be an element.