字幕表 動画を再生する
The world is entering a new age of warfare. A digital revolution is sweeping through
every military force on the planet. Leading the charge is artificial intelligence.
A technology with the power to upend everything about human conflict.
Including whether humans are involved at all.
And simmering beneath… is a global cyberwar that has already started and may never end.
Digital technology is transforming all our lives,
so no wonder it's also changing how we fight. It's making militaries smarter,
faster, more efficient. But it's also opening up the prospect of serious dangers in the future.
There's a third revolution of warfare after gunpowder and nuclear weapons
There will be more unpredictability in how we get to armed conflict, and that
will make the whole world a more dangerous place.
Here in Berlin, Germany's foreign minister warns us: a full-scale tech arms race is underway.
We're right in the middle of it. That's the reality we have to deal with.
Wir sind schon mittendrin… das ist die Realität, mit der wir es zu tun haben.
In fact critical technologies are developing so fast that SOCIETIES
can barely keep up… and ask themselves the question, is this what we want? So
in this video we're going to zero in on two risks that are not getting enough attention.
First, we'll see how a cyber intrusion against the command and control systems for nuclear weapons
could set off a terrifying chain of events.
You have to worry that it's going to escalate into something that's like
truly apocalyptic -- civilization ending.
Then we'll examine how a recent war in an obscure part of the world
provided a taste of things to come… accelerating a race for autonomous weapons.
And how the artificial intelligence behind them
could lead to conflicts that move at horrifying speed.
All of a sudden you have a have a war that no one really started,
and which could spiral out of control.
We'll catch glimpses of a future where wars can start more easily,
where they can escalate faster, and where humans can't stop them.
Machines are dictating the conduct on the battlefield.
Machines are making the ultimate decisions about life and death.
The good news is: it's not too late to make critical choices. And in the final part
we'll look at what political leaders could be doing NOW to prevent the worst from happening.
But first… we begin with a scenario that is not from the future. It could happen today.
We're going to England's north York Moors, near the coast and the windy North Sea. Here we find
what could be the most important places in the world -- that you have probably never heard of.
Its name is Fylingdales, a British air force base that's notable not for its planes, but for…
…this grey edifice, jutting out of the ground.
They call it the pyramid. But in fact, it's a giant radar.
It's not the only one.
There's something similar on the other side of the world, at Clear air force base in Alaska.
And there's another far to the south at Beale, in the heat of the California desert…
There's one hidden in the forest on Cape Cod,
Massachusetts – where America nudges out into the West Atlantic.
And in the frozen north of Greenland, far above the Arctic Circle,
you'll find that another pyramid looms.
These installations are all part of America's early warning system… powerful radars built to
detect attacks on the US homeland or American allies. Above all – incoming nuclear missiles.
It's a system that reaches out into space, where dedicated satellites keep watch from high orbit.
Constantly feeding back to the "command-and-control" apparatus
in charge of America's own nuclear weapons.
This is the nervous system of the western military alliance. It dates back to the Cold
War but in today's geopolitical tensions, it's as crucial as ever. Disrupting it could leave
the alliance blind – prone to attack. That was made clear in America's latest
nuclear posture review – essentially the instruction manual of its most powerful weapons.
This infrastructure is so important, the review said
that if it were attacked, the US might respond by using nuclear weapons.
As we're going to find out – despite their critical position at the heart of Western
security, these systems are vulnerable – to new and unpredictable threats.
The first early warning systems were built decades ago, at the height of the Cold War. Their job:
detecting nuclear missiles coming in from Russia. As they've been updated over the decades,
two crucial things have changed that make them more exposed.
First -- many are no longer focussed only on nuclear threats. They're multi-tasking.
"None of the big command and control systems whose existence has been
acknowledged by the US government are used exclusively for non-nuclear operations."
James Acton is one of the world's leading experts on nuclear security.
"That's one example of this phenomenon that I term
the growing entanglement between the nuclear and the non-nuclear domains."
This idea of "entanglement" is important. It means that the incredibly sensitive area of
nuclear weapons is no longer separated off in its own bubble. It's become mixed in with matters of
conventional warfare. And that multi-tasking means they're more likely to be a target.
In a crisis or a conflict, adversaries could have a potential incentive to attack these dual-use
command and control assets, these assets that are used for both nuclear and non-nuclear operations.
Potentially, they're doing that in order to disrupt US conventional war fighting…but that
would have the effect of degrading the US nuclear command and control architecture.
So there are more reasons to attack these targets. And on top of that comes the second big change:
they've entered the digital age, opening them up to the prospect of cyber-attack.
Systems are now relying on digital signals as opposed to analogue signals, increasingly
relying on things…like IP based operating systems IP-based operating systems: Internet Protocol
that means computer operating systems with networking capabilities
which creates vulnerabilities, for example, in the form of cyber-attacks.
Very old-fashioned nuclear command and control systems that didn't use digital
systems were invulnerable to cyber-attacks. There was no code there to do the attacking."
Today, cyber-attacks are an everyday event - we often hear about them on the news. In fact,
some say we've entered a low-grade cyber-war that will NEVER stop.
"You have a mix of state level and non-state actors constantly probing and attacking
networks around the world. That's just the reality
of 21st century life and something that we'll have to deal with."
Just about everything and everyone can be a target.
Recent attacks have hit the US government, the German parliament,
and Iran's nuclear programme. And they're just the ones that we know about.
Some of the most serious cyber-attacks have hit public infrastructure
like those against Ukraine's power grid – attacks blamed on Russia. That was so grave
that the United States stepped in to press charges against the alleged perpetrators.
No country has weaponized its cyber capabilities as maliciously and irresponsibly as Russia."
Attacks like that on civilian infrastructure have become a major public concern.
But only a small circle of experts are thinking about how a cyber-attack on
nuclear command and control systems might play out. Here, the stakes could not be higher.
To see what could happen, let's go back to the English coast and Fylingdales,
the early warning system peering across the North Sea towards Russia.
In a crisis situation with the Kremlin, this could be a prime target.
"That's so significant because that radar is the closest US
radar to Russia's biggest concentration of its nuclear forces. It's the one that would
get the quickest warning of a Russian nuclear attack. It's also the most entangled one."
Remember that idea of "entanglement" between the nuclear and the non-nuclear realms
Fylingdales is a key example of this, watching out not just for big nuclear missiles,
but also for conventional weapons.
If Russia was firing short-range ballistic missiles at Europe, Fylingdales could see
those missiles in a way that other US radars that are further from Europe couldn't.
So of all the US early warning radars, Fylingdales is the one that has the biggest Russian incentives
to attack in a crisis or a conflict. And it's the one that attacks could have the biggest effect
on in terms of degrading US strategic early warning capabilities.
And a scenario where exactly that happens is all too easy to imagine.
It's the near future. We're in Latvia, a former Soviet republic, now a member of NATO.
Protests have broken out among ethnic minority Russians,
who are accusing the government of discrimination.
As the protests turn violent, Russia begins amassing troops along the border.
Western leaders accuse Moscow of orchestrating the unrest
as a pretext to invade this tiny NATO member state.
Neighbouring Estonia and Lithuania – also former Soviet republics
and also now members of NATO, report a surge in cyber-attacks.
Fear spikes across the region.
For the first time since the Cold War, NATO and Russia are on the brink of direct conflict.
As the crisis deepens, the US detects malicious computer code planted in its
early warning networks at Fylingdales. In the heart of a system that is on ultra-high alert.
James Acton explains what happens next.
If you find malicious code in your networks. It's very hard to know what that code does. It
takes a long time to analyse the code and understand what the other side is doing.
And this makes it very hard to know whether this malicious code is just for espionage,
or is also for offensive operations as well. And in this fast-moving crisis,
the US doesn't know what the code does. It hasn't yet had a chance to analyze it even
if that code is exclusively for espionage purposes. There is a danger that the US
might conclude that it's preparations for an attack on an early warning system."
As the malware spreads, the US also has to work out who planted it. That's a process called
attribution. It takes time, and it is NOT easy. Adding pressure to the fear and deep uncertainty.
There's various countries that could have incentives to launch cyber espionage or prepare
for cyber attacks by inserting malware against the U.S. early warning system.
You know, North Korea would have an incentive for doing it. China would have
an incentive for doing it. Russia would have an incentive of doing it, maybe others, too.
Amid all that uncertainty, with the Latvian crisis ongoing, Russia becomes be the obvious suspect.
"I think there is potentially an assumption in that crisis to assume
that Russia implanted the malware. Even if you don't know for certain who did it -- Chinese
implantation or North Korean implantation -- again, in a fast moving crisis in which you
don't have time to do the attribution properly, may be misinterpreted as a Russian intrusion."
So in the heat of this crisis,
under intense pressure, the US has some enormous decisions to make.
Its most sensitive nuclear weapons infrastructure is under cyber-attack.
It doesn't know what the code is doing, or who planted it.
But the circumstances suggest it's a Russian attack. So the Americans decide to respond
in kind -- with a cyber-attack of their own against Russia's systems.
It then does the same thing against Russia…not necessarily for an attack at this point, but for
espionage and for signalling purposes and saying, you know, anything you could do, we can do better.
The problem is that Russia is very worried about the survivability of its nuclear forces.
Now Russia fears that the US is trying to mess with ITS nuclear weapons.
Discovering cyber intrusions in your command-and-control system can exacerbate
those fears. You could believe the US is preparing to attack, preparing to
eliminate the nuclear forces pre-emptively.
The two sides are entering a spiral of escalation
that leads towards disaster with a relentless logic. Russia makes the first move.
A lot of their nuclear weapons or missiles are based on trucks which they would have
to disperse to make them survivable so that the US couldn't destroy them.
So they may do that because they're worried about a US nuclear attack.
But that kind of action could confirm the US fear that they're preparing for nuclear weapon use…
and that that's the kind of scenario that I think could catalyze nuclear weapon use directly.
The US then disperses its nuclear forces that confirms Russian fears that the US
is thinking about using nuclear weapons and that leads to Russian limited nuclear use.
Limited nuclear use. We've gone from a piece of mystery code in the wrong place to a nuclear
missile launch. Let's do what the governments can't in this situation - and
slow right down to pick apart what's just happened.
Because this is how a regional crisis can turn into a catastrophic war.
In the heat of a crisis with Russia, the US detects malware in its early warning networks.
Fearing it could be Russian code aimed at disabling its systems,
it retaliates with a cyber intrusion of its own into Russia's networks.
Russia now fears its nuclear capabilities are being threatened,
and scatters its land-based weapons to avoid possible attack. When this is
picked up by the US, Washington disperses its own nuclear forces for the same reason.
Fearing an imminent nuclear attack, Russia fires the ultimate warning shot - a small nuclear
missile against a target that would result in minimal casualties, like a naval ship out at sea.
You can conceive of first use of nuclear weapons that literally kills no civilians and only a small
number of military personnel. The use of nuclear weapons against a ship at sea, far from any land,
a military vessel -- you might only kill the sailors on board that vessel and no civilians."
While the immediate damage may be limited, this crosses the threshold called "nuclear first
use. Whichever side does this, they've made the situation deadly serious.
Once you've crossed that threshold -- once nuclear first use has happened -- you have to worry that
it's going to escalate into something that's like truly apocalyptic civilization ending.
And so the real goal for me is to prevent any first use at all.
We've just seen how a cyber intrusion can escalate mercilessly into a nuclear conflict
that nobody wanted. Where things could go from THERE is the stuff of nightmares. But there
are things the world could do now to prevent such a disaster from happening in the future.
We'll look at those later. But first… let's leave this realm of
scenarios and return to the real world – and a war that has already happened.
It's late 2020 and war has broken out in a place the world had forgotten.
A festering conflict has erupted into full-scale fighting.
Ground zero is Nagorno Karabakh… a disputed region in the Caucasus mountains,
fought over by two former Soviet republics: Armenia and Azerbaijan.
This looks like a textbook regional war – over territory, over ethnic and national pride. Fought
while the rest of the world is consumed by the pandemic, it doesn't get that much media coverage.
But for those who are paying attention, it is a glimpse of future wars.
You can find it right here, in the propaganda pumping out from the start of the war.
Azerbaijan's border patrol posts this video on its YouTube account just as the conflict begins.
The lyrics are a rush of jingoistic fever, with a mantra: "hate" for the enemy.
But look carefully, and you'll see what makes this conflict a watershed in modern war.
Watch out for these trucks in the background.
In this shot you can just about see what's inside.
Then a launch, in slow motion.
What emerges is not a rocket or a missile: it has wings that are beginning to unfold
just before the video cuts away.
We can see enough to identify what this is.
It's what's called a "loitering munition" from Israel's state-owned defence manufacturer,
IAI. Its model name: the "Harop."
The company's promotional videos show what "loitering munitions" can do.
Once launched, they fly – autonomously – to a target area, where they can wait,
or "loiter" in the sky for hours, scanning for a target – typically, air defence systems.
Once they find a target, they don't drop a bomb, but fly into it, to destroy it on impact.
It's earned them the nickname "kamikaze drones."
In the war over Nagorno Karabakh,
these weapons didn't just make for good propaganda. They made a real difference.
Azerbaijan had spent years investing in loitering munitions.
Analysis by a US think tank showed that they had more than
200 units across four different models – all of them sophisticated Israeli designs.
Armenia only had a single, domestically made model with a limited range.
"The really important aspect of the conflict in Nagorno Karabakh, in my view, was the use of
these loitering munitions, so-called kamikaze drones, these pretty autonomous systems."
Ulrike Franke is one of Europe's leading experts on military drones.
They also had been used in some way or form before, but here,
they really showed their usefulness, militarily speaking, of course. It
was shown how difficult it is to fight against these systems.
As Azerbaijan celebrated victory, you could even call Nagorno Karabakh
the first war that was won -- in part -- by autonomous weapons.
Little wonder the Harop was on show that day. And other militaries were paying attention.
Since Nagorno Karabakh, since the conflict, you could definitely
see a certain uptick in interest in loitering munitions. We have seen more armed forces
around the world acquiring or wanting to acquire these loitering munitions.
The Nagorno Karabakh war amounted to a showcase
for autonomous weapons technology. With a clear message: this is the future.
It's a future that is coming at us fast. Ever more advanced models are coming onto the market…
Designed to hit a wider range of targets…
The manufacturer IAI even markets one of its models with the slogan… "fire and forget."
Fire and forget… think about that. Already, today, autonomous weapons are being used to
find a target over long distances and destroy it without human intervention.
And this revolution is just getting started – turbocharged by artificial intelligence.
In the United States, a major report from a "national security commission"
on artificial intelligence talks about AI enabling a "new paradigm
in warfighting" – and urges massive amounts of investment in the field.
This isn't all about autonomous weapons – there are many other areas of the military
which will be using artificial intelligence.
One area where we see a lot of AI-enabled capabilities is in the realm of data analysis.
We are gathering so much data in military operations. Another area, which I think is
quite promising, but also still relatively removed from the battlefield is logistics.
AI can definitely help to make this more efficient, cheaper, better, easier, all of that.
And fuelling all of this is an intensifying global competition, which spans all the
way from these more prosaic fields to the autonomous weapons we're looking at today…
The Chinese and the Russians have made it very clear that they intend
to pursue the development of autonomous weapons
Martijn Rasser, a former analyst at the CIA,
covers emerging weapons technology at Washington's leading defence think tank
and they're already investing heavily in the research and development of those systems.
It's not just the superpowers piling in. Britain's
new defence strategy also puts AI front and centre.
And as we've already seen, Israel is a leader in the autonomous weapons field.
In fact, wherever you look, countries of all sizes are jumping in.
No wonder there's talk of this becoming an arms race.
Germany's foreign minister Heiko Maas is clear that that arms race is already underway.
We're right in the middle of it. That's the reality we have to deal with.
If anything, this might go deeper than an arms race
AI is here to stay. And there is a belief among the major powers that this could make a
difference on the battlefield in the future. So they are frenetically investing in it.
Indian Diplomat Amandeep Singh Gill is the former chair of the UN government
experts' group on lethal autonomous weapons
And this is a race, in a sense, which cuts across the military and the civilian fields, because
there's also the sense that this is a multitrillion dollar question.
It's about the future of resilient economies.
That is what sets this new era apart from arms races of the past. During the Cold War,
the development of nuclear weapons was driven purely by governments and the defence industry.
Beyond power generation, there wasn't much commercial use for nuclear technology.
Today, AI is rapidly entering our everyday lives. It might even unlock the phone in your
pocket when you hold it up to your face. This emerging ubiquity of AI important.
Because it means that developments in AI cannot be contained – they
are bound to bleed across between civilian and military fields -- whether we like it or not.
AI is by definition dual use or multi use, it can be used in all kinds of ways. It really
is an enabler more than the technology. There is a whole range of applications of artificial
intelligence in the civilian realm, from health care to self-driving cars to all kinds of things.
It means that something as innocuous as a new year's celebration in Edinburgh
or St Patrick's Day in Dublin… can be powered by similar swarming technology
to what the Indian army showed off on its national day. In fact,
swarming is one of the hottest areas of autonomous weapons development right now.
The US Navy has released footage of early demonstrations.
Here, fighter jets drop over 100 tiny drones in mid-flight.
Once they're out there, it's almost impossible for the human eye to keep track of them.
The whine of their motors -- almost the only sign of the threat in the sky.
Experts say they will make highly effective weapons.
You could take out an air defense system, for example, by -- just
you throw so much mass at it and so many numbers that the system is overwhelmed.
This, of course, has a lot of tactical benefits on a battlefield. And no surprise,
a lot of countries are very interested in pursuing these types of capabilities.
Not least the head of the body advancing the US army's modernisation,
as he explained in an online think tank forum.
Most likely drone swarms are something you're going to see on the battlefield – on a future
battlefield. I don't think it's a matter of if – as a matter of fact, I think we're
already seeing some of it – it's a matter of when we begin to see it.
And feeding the momentum of this potential arms race - in order to fight these weapons,
you need these weapons. Humans don't have a chance.
When you're defending against a drone swarm, a human may be required to make that first
decision. But I'm just not sure that any human can keep up with a drone swarm.
This issue of speed gets us to a critical emerging danger of autonomous weapons...
The weapons we've seen so far are capable of a high degree of autonomy.
But they wouldn't be impossible for humans to control. Even a "fire and forget" weapon
needs a human to fire it, and they're still operating in a way that we can pretty much grasp.
Now let's think ahead, a decade or two into the future. That's a decade or two of rampant
technological development - and adoption - of increasingly autonomous weapons.
I think what is very likely that in 20 years' time we will have swarms
of unmanned systems, not even necessarily just airborne drones -- it can also be ground systems,
surface vessels, etc. So different units operating together and carrying out attacks together,
which does indeed require quite a high level of AI-enabled autonomy
To fight these systems, you will need these systems. Because human beings are simply too slow.
This is what potentially may drive an arms race that -- some actors may
be forced to adopt a certain level of autonomy, at least defensively,
because human beings would not be able to deal with autonomous attacks
as fast as would be necessary. So speed is definitely a big concern here.
And that could have fateful consequences for how wars begin.
We could find ourselves in a situation where because of this this problem of speed and
autonomous systems having to be countered by other autonomous systems, we could find ourselves
in a situation where these systems basically react to each other in a way that's not wanted
We've already seen something like this on the financial markets.
The "flash crash" of 2010 wiped more than a trillion dollars off the US stock markets
in just minutes. It was driven by trading algorithms feeding off each other
in a dizzying spiral. How it happened is STILL not fully understood.
In a flash crash, trading can be halted to prevent disaster.
The risk with a "flash war" is that there might be no pulling back.
If the beginning is bad enough, it may not even matter any more that the original event
wasn't supposed to be an attack in the first place. You could have
a situation where the counterattack is so bad that you end up in a war.
Now, think back to Nagorno Karabakh -- a regional war where autonomous weapons may
have tipped the balance. In a future world with the risk of "flash war,"
places like this could face even more instability, even more conflict.
We are moving in the world into a world where
systems will be more autonomous. But we need to make sure that we minimize the risk of
unwanted escalation, of lethality decided by machines without any human control.
But how do we do that? How do we prevent the worst? As we're about to find out
the world is struggling to find a way
We've just seen glimpses of a future that nobody could want.
Of war spinning out of control. Even erupting out of nowhere.
These are not the nightmares of science fiction. They're highly plausible outcomes of the rapid
technological development that's happening now. There is no way to stop the technologies we've
seen in this video. And we probably wouldn't want to. There are many positive applications that
will come out of them. The urgent challenge is to find ways to keep them under control.
My fear is that there will be more unpredictability
in how we get to armed conflict, so the pathways to getting to the battlefield
won't be clear to policymakers. So they will not understand fully the risks of certain
actions or certain happenings, and that will make the whole world a more dangerous place.
Amandeep Singh Gill was at the centre of United Nations efforts to try to get a
grip on autonomous weapons… a process that critics say is on the brink of failure.
This is where it all happens… The UN buildings in Geneva. It's here that delegates from UN
member states gather with experts and NGOs to talk about the future of autonomous warfare.
This process is part of what's called the UN Convention on Certain Conventional Weapons.
A diplomatic tongue-twister launched in the 1980s to try to regulate non-nuclear weapons
that were deemed so dangerous that they need special attention. Things like land
mines and blinding lasers. In 2014, lethal autonomous weapons made it onto the agenda.
It has been very slow going. The process has yielded a set of " guiding principles" – saying
that autonomous weapons should be subject to human rights law,
and that humans must have ultimate responsibility for their use.
But these "guiding principles" have no force… they're just a basis for more discussions.
For campaigners calling for a ban, that's not good enough.
We do get frustrated by the delays that have happened
and the delay in moving from discussions to actual negotiations of a new treaty.
The main problem with this forum is that it operates by consensus. So meaning any one state
can block progress and block that shift from discussions and negotiations.
Bonnie Docherty lectures on human rights at Harvard Law School - and is also a
spokeswoman for the “Campaign to Stop Killer Robots” – a high-profile coalition of NGOs.
She has mapped out principles for an international treaty.
The overarching obligation of the treaty should be to maintain meaningful human control over the
use of force, and where it should be a treaty that governs all weapons operating with autonomy
that choose targets and fire on them based on sensor's inputs rather than human inputs.
That idea of keeping "meaningful human control" is broadly echoed by many countries,
but only 30 states support the campaign. They're mostly smaller nations but include
one giant in the form of China. But Beijing's true position is blurred.
China has called for a ban on, or expressed support for a ban on USE, but has not,
to my knowledge, expressed support for a ban on development and production. We believe that
you need to prohibit development as well as use of these inherently problematic systems,
because once things are developed, the genie is out of the bottle.
And the other great military powers aren't keen at all on those sorts of limitations
either. Russia is accused by many of taking any opportunity
to thwart the Geneva talks. But there are plenty of other objectors too.
Russia has been particularly vehement in its objections… Some of the other states developing
autonomous weapon systems such as Israel, the US, UK and others have
certainly been unsupportive of a new treaty and have expressed varying degrees of support for
actually continuing discussions. So those are some of the roadblocks that we face.
As things stand, the US is highly unlikely to support a ban.
Rather, it has set out its own principles, which include human involvement.
A ban on autonomous weapons systems is essentially infeasible just because the technology
is out there. The Department of Defense has been very clear about its commitment to ethical uses
of these technologies, where right now the position is that a human being has to be on
the loop or in the loop when those weapons are used so that it won't be fully autonomous in
the sense that there won't be any human interaction with these weapons systems.
But the reality is that the US, China and Russia
are competing so intensely in all areas of AI technology that it's questionable whether any
of them would sign up to a treaty that significantly limits what they can do.
The large powers will have will always have agendas. They want
freedom of manoeuvre. They think that they need to have agency over technology development.
And sometimes they've been very sceptical of the role of international organizations,
multilateral forums in understanding and regulating technology.
Aside from the lack of interest from crucial players… the challenge of tackling an intangible
technology like AI with the traditional tools of “arms control” is genuinely difficult.
A lot of the old ways of arms control and arms control treaties
don't work anymore and don't apply anymore to these systems, because we are to put it bluntly,
we're talking about software rather than hardware. So a lot of arms control systems in the past
basically were about allocating a certain number of systems. You were allowed one hundred warheads
of this type and you were allowed one hundred heads of this type. And we're basically counting.
You can't do this with the A.I. enabled weapon systems that we were talking about,
because it doesn't matter what it looks like from the outside. But what's in there.
Germany has been quite active in trying to navigate around these problems…
its foreign minister says that the world has to find a way
Just like we managed to do with nuclear weapons over many decades, we have to forge international
treaties on new weapons technologies Heiko Maas is a member of Germany's
social democrats and has been a vocal advocate of arms control.
They need to make clear that we agree that some developments
that are technically possible are not acceptable and must be prohibited globally.
In fact the German government has laid out its intention - in the document that
underpins the current coalition. It says "We reject autonomous weapons systems
that are outside human control. We want to prohibit them worldwide."
That sounds pretty clear. But even this is complicated. Germany for instance does not
support the Campaign to Stop Killer Robots. It says there's a better way.
We don't reject it in substance –
we're just saying that we want others to be included the global controls that we would need
to ensure that autonomous weapons systems don't come into
use… So military powers that are technologically in a position
not just to develop autonomous weapons but also to use them. We need to include them.
So this isn't just a debate about the rights and wrongs of autonomous weapons.
It's also a debate about PROCESS.
On the one hand, Germany says an agreement it only worth anything if the big countries are
on board - they want that elusive consensus in the Geneva process.
On the other, the Campaign to Stop Killer Robots
says the matter is too urgent to wait. They say there's just time for one more round in Geneva.
We feel that if states don't take action by that point, that they should consider
strongly they should move outside of the Convention on Conventional Weapons and
look at other options. So they could go to the UN General Assembly to negotiate a treaty. They could
start an independent process, basically a forum that is not bound by consensus, but is
guided by states that actually are serious about this issue
and willing to develop strong standards to regulate these weapon systems.
There's precedent for this… with land mines, for example. In the 1990s,
the Geneva process couldn't find consensus.
Instead, more than 100 countries broke away to create a ban called the "Ottawa Convention.
But the great powers didn't sign. And more than 20 years later,
the US, Russia and China still haven't joined the Ottawa Convention.
It's a dilemma, isn't it? So you can do away with the rule of consensus
and then you can have results quickly, but they will not
have near universal support at the very least, they will not have support from the countries that
are developing these capabilities. But through the rule of consensus, you force those countries
to engage. So I think it's a choice that the international community makes in these forums.
So the world doesn't agree on what to do about autonomous weapons. And it can't even agree on
HOW to agree on what to do about them. In this situation, is there any prospect of a solution?
In the end we may end up with rules or norms or indeed agreements that are more focused
on specific uses and use cases rather than specific systems or technology. So
where you basically agree, for example, to use certain capabilities only in a defensive way,
or only against machines rather than humans or only in certain
contexts. But as you can imagine, implementing and, first of all, agreeing to that and then
implementing that is just much harder than some of the old arms control agreements.
Compounding this is the rock-bottom level of trust between the major powers right now.
US-Chinese talks in Alaska in early 2021 descended into a bitter round of accusations.
When there is lack of trust, you tend to attribute all kinds of intentions to the other party
and you tend to overestimate what they might be doing and overshoot in your own response.
Today, frankly, the developments on the technology front are actually adding to the mistrust.
Preventing the kind of cyber disaster we looked at earlier would REQUIRE the
great powers to cooperate. But as a first step, there are things they could do independently.
I think states need to think very carefully about how their cyber operations could be
misinterpreted, in order to fully analyze the benefits and risks before conducting
them. I think countries should adopt a rule that before launching any cyber intrusions against
nuclear command and control, including dual use assets, including the stuff that's both
nuclear and conventional, that should have to be signed off on by a Secretary of Defense or
a head of state as a way of ensuring these things are not routine.
Beyond that, the best we could hope for might be a behavioral norm
agreed between the US, Russia and China – that they would not launch cyber intrusions
against nuclear command and control systems.
The idea would be that if you detected another state in your network,
the deal was off and you could go after their network. And so
in this way, you'd hope to enforce this agreement through mutual deterrence.
But remember that problem of entanglement – of
systems involved in nuclear and non-nuclear operations.
That's going to make it potentially very difficult to define what command and control
assets are included in this kind of pledge and what command and control assets are
excluded as part of this plan. I mean, you'd have to have some pretty kind of
difficult and sensitive negotiations that I don't think states are ready for yet.
As the US, China and Russia slip deeper into an era of "great power competition,"
the challenge will be to carve out areas like this -- where they can put mutual
interest above the visceral drive to be on top. THAT is the spirit of "arms control.
You don't make arms control agreements with your best friends and allies. You always,
by definition, you know, try to negotiate them with your enemies. And this isn't
exactly new… I don't I don't think it's impossible that these players,
which are already opponents and may eventually become even more adversarial, can come together
and agree on certain minimum requirements simply because it is in everyone's interests.
For Germany's foreign minister, the whole world has responsibility here.
The world must take an interest in the fact that we're moving towards a situation with cyber or
autonomous weapons where everyone can do as they please. We don't want that.
Climate change serves as an ominous warning of what can happen when humanity
sees a common threat on the horizon but FAILS to act in time to stop it.
The Rio Summit kicked off the UN's process of talks to tackle climate change way back in 1992
It took 23 years to get to the Paris Agreement And it's clear even THAT wasn't enough
It's already too late to prevent much of the devastation
that scientists predicted from the start.
With the scenarios we've just seen -- the warning signs are just as clear,
and if anything even more urgent.