In linear algebra and other types of mathematics, the notion of perpendicular is generalized to "orthogonal" in different coordinate systems, with the idea that a certain type of "product" between two vectors (i.e. the "dot product" in our familiar cartesian coordinates, and more generally the "inner product") equals 0.

At a small enough scale we can treat the earth's surface as a flat plane and still get reasonable results, though on bigger scales we have to use a roughly spherical geometry. On a sphere you can still have two orthogonal axes for your coordinate system, just the path you travel affects what direction you end up facing differently from a plane. Triangles add up their angles differently from a flat plane, etc., but that only matters if you are traveling very far.

So there is one broader mathematical definition of perpendicular of which our coordinates on Earth are a special case. But why have we chosen perpendicular axes? Historically, perhaps because it's a simple and intuitive way of describing the geometry we can see and interact with. Sort of an emergent property of human thinking, when people in antiquity were working mathematics as it related to everyday experience. Humans seem to like symmetry and simplicity, so a four-fold symmetry of right angles is a "nice" coordinate system to us.