"Go to, let us go down, and there confound their language, that they may not understand one another’s speech."
This confounding did not consist of splitting one language into multiple languages, and thus rendering only small groups comprehensible to each other. When the Gods confounded the language, it was by replacing the pure Adamic language with a more fallen one, which failed to reliably express meaning the way the old language had. And to this very day, our language is still such that two people speaking the same language often only believe that they are actually understanding one another.
(I have a strong testimony of at least the last sentence of this midrash.)