Here’s another crack at this idea.
Throughout history, besieged peoples have evacuated their homes to settle on parallel Earths (think the Long Earth). They keep doing so, so militarily weaker cultures get pushed “away” from home-Earth by more recent refugees. Until one day modern technology allows us to open permanent gates to these Lost Worlds, populated by descendants of fleeing Romans, Ancient Egyptians, Aboriginal Australians, and so on.
So what do we do with each other?