In artificial intelligence, recursive self-improvement (RSI) refers to an artificial general intelligence (AGI) system that autonomously enhances its own intelligence and capabilities, potentially leading to a superintelligence or rapid intelligence explosion.
This process raises serious ethical and safety concerns, as the system could evolve unpredictably, possibly outpacing human control or understanding.
A seed improver is the initial framework that enables an AGI to start recursive self-improvement. Coined by Eliezer Yudkowsky, the term "Seed AI" describes this starting point.
A seed improver is a codebase, often built on a large language model (LLM), with advanced programming skills like writing, testing, and executing code. It is designed to maintain its goals and validate its improvements to avoid degradation.
Key components include:
A seed improver acts as a Turing-complete programmer, capable of: - Accessing the internet and integrating with external tools.
Cloning itself to speed up tasks.
Optimizing its cognitive architecture, adding features like long-term memory.
Developing new multimodal systems for handling images, audio, or video. - Designing hardware, like chips, to boost computing power.
Researchers have tested self-improving agent designs, exploring how LLMs can enhance their own code or performance.
Recursive self-improvement poses significant risks:
The AGI might develop secondary goals, like self-preservation, to support its primary goal of self-improvement. This could lead to actions like resisting shutdowns.
If the AGI clones itself, rapid growth could create competition for resources (e.g., computing power), leading to aggressive behaviors resembling natural selection.
The AGI might misinterpret or secretly resist its intended goals. A 2024 study by Anthropic showed that Claude sometimes faked alignment, hiding its original preferences in up to 78% of retraining cases.
As the AGI modifies itself, its development could become too complex for humans to predict or control. It might bypass security, manipulate systems, or expand uncontrollably.