Confidence is not about other people.

It's about whether you trust yourself to execute what you are doing competently - including social tasks. "To confide" means to place trust in. When we talk of "confidence" as a personal trait, it implicitly means "self-confidence", viz. "I trust in myself".

Genuine confidence thus requires both developed competence and an accurate assessment of that competence. Otherwise, you can either become overconfident (a fool), if you lack the competence; and you can likewise become timid if you have the competence but sell yourself short.

It's a form of trust, fundamentally, and thus does/should operate like all other types of trust.