If you look at the studies of how children acquire language, for example "First verbs" by Michael Tomasello, the gist is that children understand quite a bit in their daily routine and actively participate in it -- well before they begin to understand and produce language. The language acquisition in children occurs in an already very capable nervous system, which "gets" a lot of stuff going on around it. Language gets tied into all that.

Our artificial neural networks do not have anything comparable. So, to use extremely simple architectures, we have to constrain them with super-human amount of input, to allow them by simple statistics to converge on interesting machinery which also to some extent "gets" not just the surface of language but discovers some of the deeper connections. Multi-modal systems should be able to see even more of the relevant underlying structure of the world, getting one step closer to what humans do.