I think you miss the central point - that our main enemy is death, and that we can't stop death without nanotech and AI (but their power may need to be limited to a level at which they can solve aging while still remaining controllable).
You can't say that death is OK but extinction is not, because both are death, only on different levels. If death becomes potentially optional, we can't take that choice away from humans by limiting life-extension technologies.
Death is the ultimate enemy, and it needs to be prevented for all humans (and maybe even for all living and possible beings).
We also can't pretend that new supertech will never appear.
Fighting death and stopping aging may look like one of the forms of human augmentation.
And people are interested in digital minds not because digital minds are somehow good per se, but because they don't see any other way to immortality. Digital minds are copiable and archivable, so they are almost eternal.
Compared with preventing death, all other transformations into posthumans are small cosmetic changes. This is the core idea of what Fedorov called the Common Cause.
So here is the real problem: how to solve death for everybody but remain biological humans? It is just a failure of imagination that we can't find the answer. There are several ideas:
1.Nanotech augmentation of biological bodies: bodies remain biological, and nanotech only cures aging and damage. Current vaccines and antibiotics are primitive forms of this approach.
2.Biouploading - the brain's information is transferred to a new blank biological brain (cloned or grown as an organoid). This should be done only if nanotech augmentation, discussed above, fails. There is an even stronger form of biouploading, which preserves biological identity: in it, most of the neurons of a living person are moved into a mesh or into another blank brain. More here: https://www.lesswrong.com/posts/zHnFRiaQn28ApEhTH/biouploading-preserving-my-living-neurons-and-connectome-as
3.Cheap cryonics via chemical preservation of the brain.
4.Sideloading as a temporary solution before we have biouploading.
Most of these ideas are cheap, affordable, and do not change human biological nature (biouploading is more complex, though, and would become cheap only with self-replicating bionanotech). Humans can continue to have sex and reproduce in normal ways, and only in older age use these things as protection against aging.
Here I think we have to distinguish between life extension and capabilities that give power over other people. The second type of capability has existed since the time of stones and bows, and is getting even more powerful over time.
The argument that only rich people will use the tech and will limit it for poor people does not work, because of induction. If Elon Musk gets indefinite life extension, his wife will ask for the same. If the wife gets it, her mother will ask for the same, and so on. The cutoff surface will grow, and more people will demand it from those who can't deny it to them.
One more reason for cheaper biotech is illegal copies and generics. There is a large market for generics in India. Russia under sanctions started copying expensive gene-therapy solutions. Most attempts to keep a moat around some secret technology have failed historically.
The argument that there is no better phone in the world than the iPhone and almost everybody can afford it also works - this is a result of economies of scale. And the biggest problem with life-extension tech is testing, which requires large-scale clinical trials with millions of people participating.
Note that high-status people need low-status people to feel superior to them. If other people go extinct, the high-status people will have ordinary status. Status games are unpleasant, but they are also protective.
Someone becomes powerful anyway - dictators have existed since the time of Ancient Greece, and they were cruel and made bloody mistakes (why the war for Troy?). So limiting capabilities and intelligence will not prevent power concentration, which is a natural game-theoretic outcome (and Marx showed this, if you want a left-wing solution). Sure, dictators die, but dynasties are stable — there were about 30 dynasties in ancient Egypt. North Korea is stable despite the Kims' deaths. Also, Russian history shows that most rulers who died of aging had, on average, less than 4 years left before the coup that would have replaced them. So immortality will not protect them. Whether it is possible to create a stable system of government without coups and without democratic elections is an open question.
The replacement argument based on comparison with monkeys doesn't work: after a steep decline in the horse population in the US following the invention of the car, the horse population grew again, as horses are used for leisure and as pets, and has now reached 7 million. There are billions of cats, which have flourished together with humans. Dogs' intelligence is still growing via selection. Most monkeys are killed by low-intelligence poachers or by habitat destruction. So the biggest risk comes from the nearest competitors, and a more advanced society can get what it wants without destroying - it even passes nature protection laws, etc.
In this framework, space exploration becomes not a colonisation experiment but a solution to the overpopulation problem (which does not depend on life extension per se, as even a short-lived human population can grow exponentially if people have about 3 kids each. Limiting the number of children and life expectancy in no way sounds like flourishing. If lifespan is unlimited, then having many children over a lifetime is not a problem, as they can appear one after another, after long intervals, when new resources appear for them).
The second problem is that humans are not just one more animal species that we can preserve indefinitely in the same state. Humans are the pinnacle of evolution and are interested in constant development and in being something else. Human nature is built around becoming someone else, moving from one stage to another. A child grows into an adult, a soldier dreams of being a general, a migrant searches for a better life.
"Human flourishing" without any perspective is a jail. We are not plants to flourish. And improving our own minds, getting indefinite life extension, and exploring space give us such a perspective.
There are obvious failure modes: replacing everyone with philozombies, or creating several very powerful non-human entities that will eat everyone else. I think you are correct to identify them, but it is also not surprising that most "TESCREAL" people would agree that such outcomes are bad.
Do we need ASI for this? I think immortality can be solved with an AI at IQ=200, maybe 300 to get advanced nanotech quickly. There is a trade-off between the maximum AI level and the speed at which we get life extension – but also the risk of losing control. So it is better to limit AI to the IQ=200 level: good enough for most useful things, but relatively safe. Any AI more capable than that is needed only for war with other AIs. (A benevolent AI singleton that cares about people's values is one of the possible stable solutions, though that is a different story. If you want another success-without-AI story, look at the Noon Universe by the Strugatskys: https://en.wikipedia.org/wiki/Noon_Universe.)
E. Torres wrote a popular essay that TESCREAL people are all pro extinction of current human species. https://www.realtimetechpocalypse.com/p/a-pro-human-manifesto-part-1
Here is my replay:
I think you miss the central point - that our main enemy is death, and that we can't stop death without nanotech and AI (but their power may need to be limited to a level at which they can solve aging while still remaining controllable).
You can't say that death is OK but extinction is not, because both are death, only on different levels. If death becomes potentially optional, we can't take that choice away from humans by limiting life-extension technologies.
Death is the ultimate enemy, and it needs to be prevented for all humans (and maybe even for all living and possible beings).
We also can't pretend that new supertech will never appear.
Fighting death and stopping aging may look like one of the forms of human augmentation.
And people are interested in digital minds not because digital minds are somehow good per se, but because they don't see any other way to immortality. Digital minds are copiable and archivable, so they are almost eternal.
Compared with preventing death, all other transformations into posthumans are small cosmetic changes. This is the core idea of what Fedorov called the Common Cause.
So here is the real problem: how to solve death for everybody but remain biological humans? It is just a failure of imagination that we can't find the answer. There are several ideas:
1. Nanotech augmentation of biological bodies: bodies remain biological, and nanotech only cures aging and damage. Current vaccines and antibiotics are primitive forms of this approach.
2. Biouploading - the brain's information is transferred to a new blank biological brain (cloned or grown as an organoid). This should be done only if nanotech augmentation, discussed above, fails. There is an even stronger form of biouploading, which preserves biological identity: in it, most of the neurons of a living person are moved into a mesh or into another blank brain. More here: https://www.lesswrong.com/posts/zHnFRiaQn28ApEhTH/biouploading-preserving-my-living-neurons-and-connectome-as
3. Cheap cryonics via chemical preservation of the brain.
4. Sideloading as a temporary solution before we have biouploading.
Most of these ideas are cheap, affordable, and do not change human biological nature (biouploading is more complex, though, and would become cheap only with self-replicating bionanotech). Humans can continue to have sex and reproduce in normal ways, and only in older age use these things as protection against aging.
Here I think we have to distinguish between life extension and capabilities that give power over other people. The second type of capability has existed since the time of stones and bows, and is getting even more powerful over time.
The argument that only rich people will use the tech and will limit it for poor people does not work, because of induction. If Elon Musk gets indefinite life extension, his wife will ask for the same. If the wife gets it, her mother will ask for the same, and so on. The cutoff surface will grow, and more people will demand it from those who can't deny it to them.
One more reason for cheaper biotech is illegal copies and generics. There is a large market for generics in India. Russia under sanctions started copying expensive gene-therapy solutions. Most attempts to keep a moat around some secret technology have failed historically.
The argument that there is no better phone in the world than the iPhone and almost everybody can afford it also works - this is a result of economies of scale. And the biggest problem with life-extension tech is testing, which requires large-scale clinical trials with millions of people participating.
Note that high-status people need low-status people to feel superior to them. If other people go extinct, the high-status people will have ordinary status. Status games are unpleasant, but they are also protective.
Someone becomes powerful anyway - dictators have existed since the time of Ancient Greece, and they were cruel and made bloody mistakes (why the war for Troy?). So limiting capabilities and intelligence will not prevent power concentration, which is a natural game-theoretic outcome (and Marx showed this, if you want a left-wing solution). Sure, dictators die, but dynasties are stable — there were about 30 dynasties in ancient Egypt. North Korea is stable despite the Kims' deaths. Also, Russian history shows that most rulers who died of aging had, on average, less than 4 years left before the coup that would have replaced them. So immortality will not protect them. Whether it is possible to create a stable system of government without coups and without democratic elections is an open question.
The replacement argument based on comparison with monkeys doesn't work: after a steep decline in the horse population in the US following the invention of the car, the horse population grew again, as horses are used for leisure and as pets, and has now reached 7 million. There are billions of cats, which have flourished together with humans. Dogs' intelligence is still growing via selection. Most monkeys are killed by low-intelligence poachers or by habitat destruction. So the biggest risk comes from the nearest competitors, and a more advanced society can get what it wants without destroying - it even passes nature protection laws, etc.
In this framework, space exploration becomes not a colonisation experiment but a solution to the overpopulation problem (which does not depend on life extension per se, as even a short-lived human population can grow exponentially if people have about 3 kids each. Limiting the number of children and life expectancy in no way sounds like flourishing. If lifespan is unlimited, then having many children over a lifetime is not a problem, as they can appear one after another, after long intervals, when new resources appear for them).
The second problem is that humans are not just one more animal species that we can preserve indefinitely in the same state. Humans are the pinnacle of evolution and are interested in constant development and in being something else. Human nature is built around becoming someone else, moving from one stage to another. A child grows into an adult, a soldier dreams of being a general, a migrant searches for a better life.
"Human flourishing" without any perspective is a jail. We are not plants to flourish. And improving our own minds, getting indefinite life extension, and exploring space give us such a perspective.
There are obvious failure modes: replacing everyone with philozombies, or creating several very powerful non-human entities that will eat everyone else. I think you are correct to identify them, but it is also not surprising that most "TESCREAL" people would agree that such outcomes are bad.
Do we need ASI for this? I think immortality can be solved with an AI at IQ=200, maybe 300 to get advanced nanotech quickly. There is a trade-off between the maximum AI level and the speed at which we get life extension – but also the risk of losing control. So it is better to limit AI to the IQ=200 level: good enough for most useful things, but relatively safe. Any AI more capable than that is needed only for war with other AIs. (A benevolent AI singleton that cares about people's values is one of the possible stable solutions, though that is a different story. If you want another success-without-AI story, look at the Noon Universe by the Strugatskys: https://en.wikipedia.org/wiki/Noon_Universe.)