If a work has a particular set of qualities that permeate its theme and those qualities are unique compared to any existing genre, then you have created a genre. It's also, I think, about quantity. One work that is unique is quirky. A long series of works by many different authors with the same themes is a genre.

If you created a book about a world run by spiders (which evolved that way and not by technology) and humans as annoying, scary little creatures and then other authors, filmmakers and artists started doing the same with the repeated trope of a spider-run planet, that would eventually become a genre. Some will say that it's science fiction but it's not. Speculative fiction? Sure. But speculative fiction transcends genre whether it's alternative history, fantasy, science fiction, litfic or whatever.