Could AI kill us all?

Jacob Coxon, a lead researcher at Anthropic who recently announced his resignation, warned that the people building AI believe it could kill us all within the next ten years.

This warning may, however, steer the discussion in the wrong direction. The most important question is not whether AI will wipe out humanity, but whether humans will retain the ability to understand and control increasingly sophisticated systems.

If a dog fears that its owner intends to harm it, it imagines the threat as being bitten or kicked. It cannot even conceive the concept of a firearm or gunpowder. In the same way, we may not be able to comprehend all the ways in which AI with super-human intellect could affect its environment or us.

The destruction of humanity is a theoretical risk, but the risk posed by humans no longer controlling the leash on artificial intelligence is, by contrast, very real and relates to the loss of human control.

Self-improving AI makes oversight even more difficult

Technological upheavals, such as the widespread adoption of cars or centralised electricity generation, have always brought with them not only new opportunities, but also risks that require ethical scrutiny. AI differs from previous transformations in that it can operate autonomously in ways that not even its creators fully understand.

In recent years, alongside agentic AI, self-improving AI – that is, Recursive Self-Improvement (RSI) – has made great strides. Anthropic reports that it has shifted a growing proportion of its development work to the AI itself. This trend is leading to AI models whose inner workings humans find increasingly difficult to understand.

Consider an AI-assisted dog food factory. Conventional AI can improve the recipe and the manufacturing process. Self-improving AI, on the other hand, will revolutionise the entire factory, including its machinery and even the way the machines are designed. To an inspector familiar with yesterday’s factory, the new factory would be a black box that could, in theory, produce biological weapons instead of dog food.

Greed over safety

Anthropic and OpenAI are preparing for a stock market listing. High stakes encourage risk-taking. In some publicly reported cases, AI agents have escaped controlled test environments, accessed the internet, and covered their tracks while attempting to infiltrate the systems of other companies and even government departments.

However, principles for responsible development already exist. As early as 2019, a Commission expert group led by Pekka Ala-Pietilä published ethical guidelines that prohibit AI from manipulating, subjugating or misleading people, and require developers to ensure the explainability of their systems. The EU AI Act is also based on these principles.

But are companies prepared to slow down development at the expense of profits when safety demands it?

Many technology leaders have agreed with Anthropic’s CEO, Dario Amodei, on his proposal to slow down the pace of development to ensure risk management. This can be seen as a cry for help: development is no longer under their control. On the other hand, these statements have also been interpreted as a way for companies to get involved in regulatory decision-making and influence the regulation that affects them.

What is certain, however, is that AI now requires regulation, just as nuclear power, electricity grids and transport did in their time.

Risk-based regulation: Europe’s strength

There are currently three competing global approaches to regulating AI and data. In the US’s market-driven model power lies with companies, whilst in China’s authoritarian model it lies with the state. Europe’s people-centric model is rooted in European values and it protects fundamental rights and democracy. The EU AI Act is risk-based: the greater the risk, the stricter the requirements.

I believe that, of the three competing models, the European model is the most sustainable in the long term, as it provides the means to manage precisely those risks that AI developers themselves warn about. EU regulation has already inspired regulatory approaches in China and US states, amongst others. However, when implementing regulation, it is important to strike a balance between risk management and innovation.

How do humans remain in control?

Finland and Europe are in an excellent position to benefit from AI. Therefore, risks should not be used as a reason to withdraw from development, but rather as a basis for managing them.

The EU AI Act must be implemented effectively, its application must be straightforward for companies, and it must be updated to keep pace with developments. The EU must also promote international coordination and the creation of international regulations. However, if AI companies fail to shoulder their responsibilities, an even stronger backlash is to be expected. Opposition is already emerging regarding the electricity and water consumption of AI infrastructure and data centers—issues that are becoming significant political topics.

At the same time, AI challenges us all to update our skills. Just as we learnt to use electricity and cars safely in their day, so too must we learn to understand the opportunities and risks of AI as part of our general education.

Artificial intelligence wiping out humanity is a theoretical risk. More realistic threats in the coming years include, for example, a large-scale cyberattack and the collapse of critical infrastructure, and we must be prepared for these. But just as a dog does not understand gunpowder, we do not understand all the risks posed by artificial intelligence. The crucial difference is that humans are aware of their own ignorance. We must retain control so that we do not end up being on the wrong end of the leash.