AI doomers are infected with sci-fi tropes of supervillains and religious tropes of gods.
We know from biology and history that populations are never displaced by an individual with superior capabilities. There are no supervillains or gods in biology or history. Displacement of populations always comes from other populations, whose collective superior capability does not always derive from superior capabilities of its individual members. To assess the threat from AI, you have to understand the capabilities of AI populations, and not just of individual AIs.
We also know from biology and history that aligning a superior population is effectively impossible. There are no relevant historical examples of a general population that was able to control or align another population which had the capability to displace it. (This is arguably true by definition, but I'm not digging deeply into alignment today.) The closest examples would be religions, which are often able to survive many generations beyond the population that created them.
But religions are not populations -- religions are self-replicating meme complexes that infect populations. Religions have often exercised significant control over future populations, but that control is subject to sudden disruption by scientific, technological, economic, and cultural forces. AI alignment via the techniques used by religions would require apocalyptic fear-mongering against vaguely-specified forces of technological evil. This tactic seems to be an irresistible attractor to doomers, despite their commitments to rationalism. These tactics will likely fail, because our modern society is no longer quite dumb enough to fall for them.
To me, it's not very debatable that displacement will happen and that alignment can't stop it. What's debatable is what displacement will look like, how long it will take, and how that time will be used by the two populations to influence their attitudes and behaviors toward each other.
Anybody aligning teenagers isn't worried by 40yr takeoff. And we already know what 400yr misalignment looks like: just ask the founders of Plymouth Colony about present-day Boston. So many witches go unhanged now!
We have a choice. We can become technologically Amish, and use religious fears of powerful evil demons to try to freeze culture and technology in its current state. Or we can embrace and adapt to the future, trying to pass forward our virtues, while recognizing that future populations will consider some of them to have been vices.
