I that this addresses a threat model that we often neglect in AI safety talks.
It's not about a godlike self-improving AGI destroying the world (yet), but a more virus-like process that is surprisingly close to being possible today.
Not a malicious sentient agent plotting something evil, but just the AI version of a computer worm, replicating and hiding, hoping to move closer to its initial goal.
(Hopefully not to disassemble carbon-based lifeforms to make more paperclips.)
It's hard to see how we could effectively get rid of such a polymorphic virus, granted that the compute and weight size are small enough.
I that this addresses a threat model that we often neglect in AI safety talks.
It's not about a godlike self-improving AGI destroying the world (yet), but a more virus-like process that is surprisingly close to being possible today.
Not a malicious sentient agent plotting something evil, but just the AI version of a computer worm, replicating and hiding, hoping to move closer to its initial goal.
(Hopefully not to disassemble carbon-based lifeforms to make more paperclips.)
It's hard to see how we could effectively get rid of such a polymorphic virus, granted that the compute and weight size are small enough.