Yes, second language learning difficulty is measured relative to the first language.

The US DLI and state department difficulty rankings are based on the number of hours for an English speaker to learn the target language; it's different for speakers of other languages. For instance Korean speakers have little trouble with Japanese, despite these being category IV (hardest) on the DLI scale.

Factors that make languages easy or hard are vocabulary, grammar, reading and writing system, and cultural cues (and a few others that I can't recall at the moment). Category IV languages have differences from English in all of these categories.