There's some evidence that this is what makes learning a language with a very different set of phonemes more difficult as an adult. Babies have a developmental stage where they babble all sorts of sounds--pretty much anything our mouths/voices can make--then those sounds quickly get whittled down to the phonemes and tones we hear being spoken around us. The distinctions between certain sounds get washed out, and the barriers between others are established, and you continue the practice and muscle memory of making the sounds in your native language. If you're a baby in an English speaking country, a front-of-the-mouth "oo" sound is the same as a back-of-the-mouth "oo" sound (e.g. "duuuuude" vs "cooooool"). If you're a baby in a French speaking country, those are two distinct sounds that matter to your ability to understand words. That sound difference is as ingrained to someone raised speaking French as the difference between the "i" sound in "ship" and the "ee" sound in "sheep" is to an English-speaker--a sound difference that doesn't exist in the French language because it only has one close front vowel. Obviously it's not impossible to learn new phonemes as an adult, but babies who are raised bilingual are almost always going to have an ingrained level of phonetic fluency in the other language that's difficult for an adult to achieve.

If the topic is something you're interested in, there's a podcast episode by linguists Lauren Gawne and Gretchen McCulloch that covers this https://www.youtube.com/watch?v=9l-73oOgbmo