Autonomous Weapons Systems Dr Gilles Giacca Arms Unit, ICRC Legal Division

1. Autonomous Weapons Systems Dr Gilles Giacca Arms Unit, ICRC Legal Division

2. Definitions and terminology • lethal autonomous weapon systems (LAWS), fully autonomous weapon systems, killer robots, etc. • no international agreed definitions

3. Terminology • Human-controlled (“human-in-the-loop”) systems: Can perform tasks delegated to them independently (navigation, systems control, target detection, weapons guidance) but depend on certain real-time commands by a human operator (e.g. current drones). • Human-supervised (“human-on-the-loop”) systems: Can conduct targeting process independently, but remain under the real-time supervision of human operator who can override robotic decisions. • Autonomous (“human-out-of-the-loop”) systems: weapons that can independently select and attack targets, that is, they have autonomy in the ‘critical functions’ of acquiring, tracking, selecting and attacking targets. • “Automated”: Restricted to predefined & controlled environment. • “Autonomous”: Can operate in open & unpredictable environment.

4. Autonomous weapon systems • Independently search for, identify, and attack targets • Autonomous functions of identifying, tracking, selecting and attacking targets, i.e. ‘critical functions’ • Concern is loss of human controlover use of force • Compliance with international humanitarian law (IHL) • Ethical concerns • What is meaningful or adequate human control?

5. Air defence weapons • Fixed on ships, ground installations, vehicles. • Targets: rockets, mortars, missiles, aircraft, drones, boats. • Autonomous modes for identifying, tracking, selecting and attackingtargets.

6. Loitering munitions and certain missiles • Targets: humans, vehicles, aircraft, objects. • Launched by a human operator. • Then autonomousidentifying, tracking selecting and attacking targets.

7. Armed drones? • Targets: humans, vehicles, aircraft, objects. • Autonomousidentifying and tracking of targets? • Remote controlled selecting and attacking targets. • Autonomous attack in the future?

8. Anti-personnel ‘sentry’ weapons • Targets: humans. • Autonomous modes for identifying, tracking and selecting targets. • Remote controlled or autonomous for attacking targets.

9. Mobile ground weapon systems? • Targets: humans, vehicles, aircraft, objects. • Autonomousidentifying and tracking of targets? • Remote controlled selecting and attacking targets. • Autonomous attack in the future?

10. Mobile surface water and underwater weapon systems • Targets: humans, boats, submarines. • Surface: Remote controlled selecting and attacking targets. Autonomous in the future? • Underwater: Human launch. Then autonomousidentifying, tracking, selecting and attacking targets.

11. Legal questions • Legal review of new weapons (Article 36 API) • Realistic assessment of weapon and intended use • Are there any doubts about IHL compliance • Distinction? • Proportionality? • Precautions in attack? • Accountability? • Human control and judgement implicit in IHL? • Predictability and reliability?

12. Ethical concerns • Principles of humanity and dictates of public conscience (Martens Clause) • Can we allow machines to make life and death decisions? • Does it matter how you die? (Human dignity) • Who is responsible?

13. Technical characteristics, performance and operational parameters • The task the weapon system is being used for (e.g. offensive or defensive operations); • The type of target it attacks (e.g. objects and/or personnel); • The type of force it is using (e.g. kinetic or non-kinetic) and type of munitions (e.g. bullets or explosive weapons, blast radius, etc); • The environment in which it is used (e.g. air, ground, or sea; simple or “cluttered” environments); • The mobility of the weapon in space (e.g. stationary or mobile; narrow or wide geographical area); • The time-frame of the action of the weapon (e.g. short or long time periods); • The nature of the operation (e.g. whether the attack is planned in advance v. time-sensitive targets) • The level of human supervision of the weapon (e.g. supervised or unsupervised; ability to deactivate or not).

14. Future international discussions CCW, three informal meetings of experts on “lethal autonomous weapon systems (LAWS)” - April 2014, - April 2015 - April 2016 A proposal to establish a Group of Governmental Experts (GGE) for future work in 2017/18 will be considered at the Review Conference in 12-16 December 2016.

15. Future international discussions • Meaningful, effective and adequate human control should be maintained. • Common understanding needed of the characteristics of an autonomous weapon system. • Examine autonomy in “critical functions” of existing and emerging weapon systems. • Use to develop deeper understanding of legal and ethical questions. • Setting limits on autonomy?

16. Thank you! Contact: ggiacca@icrc.org

17. What does IHL say about autonomous weapons? • Law of Weaponry and of Targeting apply unchanged (those who plan/decide upon) • Remoteness not prohibited/restricted per se • Article 36, API binds most states –weapons, means and methods + study, development, acquisition and adoption • Customary rule to review new weapons

18. Technical issues • Focus on weapon systems with increasing autonomy in their ‘critical functions’ • Do not need to be complex (e.g. artificial intelligence) to raise concerns • Some concerns may be context dependent • Environment? Task? Target? Time of action? Mobility? Supervision? Predictability? Reliability? • Is there meaningful or adequate human control over the use of force? • Examples …

21. ICRC working definitions Autonomous weapon system Any weapon system with autonomy in its critical functions. That is, a weapon system that can select (i.e. search for or detect, identify, track, select) and attack (i.e. use force against, neutralize, damage or destroy) targets without human intervention.

25. Is there a requirement for human judgment/control in the targeting phase? • All rules of IHL are addressed to human beings/parties to the conflict • IHL does not foresee that compliance is ensured by a weapon system operating in a vacuum • Compliance is achieved by the manner in which the weapon is employed in a particular situation

26. The principle of distinction Article 48 of the First Additional Protocol: 'In order to ensure respect for and protection of the civilian population and civilian objects, the Parties to the conflict shall at all times distinguish between the civilian population and combatants and between civilian objects and military objectives and accordingly shall direct their operations only against military objectives.'

27. The principle of proportionalityArt. 51(5) AP I & CIHL 14 • ‘Launching an attack which may be expected to cause incidental loss of civilian life, injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the concrete and direct military advantage anticipated, is prohibited.‘

28. The main IHL practical problem envisaged in the future? • The main practical problem relates to the application of the principle of proportionality.  How will machines – unassisted by human beings – be capable of assessing in a concrete case whether collateral damage to civilians/civilian objects is expected to be “excessive” compared to the anticipated military advantage?

29. Precautions in attackArt. 57 AP I & CIHL 15-21 • Article 57 requires ‘those who plan for or decide upon an attack’ to (for example) ‘take all feasible precautions in the choice of means and methods of attack with a view to avoiding, and in any event to minimizing, incidental loss of civilian life, injury to civilians and damage to civilian objects.’

30. The notion of “meaningful human control” There seems to be general recognition that human beings must retain some degree of control over the ‘critical functions’ of weapon systems and the use of force. • Need for a greater clarity on what is meant by ‘meaningful human control’; • Is the required level of human control the same in all circumstances? And, how will the concept of ‘meaningful human control’ be implemented?

31. 3 main stages involved Stage 1. the development of the weapon system, including its programming; Stage 2. the deployment and use of the weapon system, including the decision by the commander or operator to use or activate the weapon system; Stage 3. the operation of the weapon system during which it selects and attacks targets.

32. 3 main stages involved Stage 2. the deployment and use of the weapon system, including the decision by the commander or operator to use or activate the weapon system; • To be considered in light of the decision-making (targeting) process: • Overall Goals • Target analysis and data collection • Validation • Capability analysis and assignment based on means and method available • Mission planning and execution

33. Technical characteristics, performance and operational parameters • The task the weapon system is being used for (e.g. offensive or defensive operations); • The type of target it attacks (e.g. objects and/or personnel); • The type of force it is using (e.g. kinetic or non-kinetic) and type of munitions (e.g. bullets or explosive weapons, blast radius, etc); • The environment in which it is used (e.g. air, ground, or sea; simple or “cluttered” environments); • The mobility of the weapon in space (e.g. stationary or mobile; narrow or wide geographical area); • The time-frame of the action of the weapon (e.g. short or long time periods); • The nature of the operation (e.g. whether the attack is planned in advance v. time-sensitive targets) • The level of human supervision of the weapon (e.g. supervised or unsupervised; ability to deactivate or not).

34. Weapons Review In the study, development, acquisition or adoption of a new weapon, means or method of warfare, a High Contracting Party is under an obligation to determine whether its employment would, in some or all circumstances, be prohibited by this Protocol or by any other rule of international law applicable to the High Contracting Party.Art.36, Additional Protocol I

35. Key questions • Why legal reviews are important? • What is the scope of the review and how should it be done? • What are the specific challenges posed by autonomous weapon systems?

36. Article 84, Additional Protocol I Rules of application: “The High Contracting Parties shall communicate to one another … their official translations of this Protocol, as well as the laws and regulations which they may adopt to ensure its application.”

37. What must be reviewed?What are "new" weapons? • weapons of all types • weapons acquired for the first time • not necessarily ‘new’ in a technical sense • modified weapons • all weapons developed or acquired subject to new treaties

38. What is the scope of the review and how should it be done?

39. Legal Reviews of Weapons • Legal requirement for States party to AP I • Formal standing mechanism or procedures • Apply multidisciplinary approach to emerging technology (AWS, cyber, etc) • Makesgood policy sense: • ensures that the State is capable of respecting its international legal obligations in armed conflict • especially important in light of rapid development of new weapons technologies • Update of the ICRC Guide, 2016

40. Weapons Review: selected questions • How does the reviewing authority attach conditions/restrictions on a new weapon (e.g. rules of engagement, standard operating procedures, or instructions)? • What types of weapons are reviewed under your national system? Are there any weapon types that they do not review and if so why? Are [means and] methods of warfare covered by the review mechanism? Do you legally review new military doctrine? • What kind of information does the weapon reviewer consider and from which sources? What experts, other than legal experts, are involved in the review process? • Does the legal review take into account possible future trends in the development of IHL?

41. Weapons Review: selected questions • What are the challenges, if any, to the conduct of legal reviews of certain new technologies of warfare (cyber technology, automated/autonomous weapons systems, etc)? • How are ‘targeting rules’ (e.g. distinction, the rules of proportionality and precautions in attack) considered in reviewing the weapons? Please explain • How is the reliability of the weapons tested? Has your State developed any standard on testing and validation? What is the level of reliability considered to be acceptable? • Does the review mechanism undertake common testing with other States? • Are weapons already fielded in the battlefield subject to regular review based on its technical performance?

42. Future international discussions CCW, three informal meetings of experts on “lethal autonomous weapon systems (LAWS)” - April 2014, - April 2015 - April 2016 A proposal to establish a Group of Governmental Experts (GGE) for future work in 2017/18 will be considered at the Review Conference in 12-16 December 2016.

43. Future international discussions • Meaningful, effective and adequate human control should be maintained. • Common understanding needed of the characteristics of an autonomous weapon system. • Examine autonomy in “critical functions” of existing and emerging weapon systems. • Use to develop deeper understanding of legal and ethical questions. • Setting limits on autonomy?