Forbidden Arms: The Case Against Autonomous Mass Destruction

In 1925, the Geneva Protocol banned the use of chemical and biological weapons due to their indiscriminate and inhumane impacts. Nearly a century later, a new generation of weapons employing artificial intelligence (AI) poses similarly grave threats.

Autonomous and semi-autonomous weapons utilizing AI risk fueling arms races, empowering non-state actors, lacking human judgment and control, and enabling mass disruption and destruction. While AI promises societal benefits, its weaponization demands urgent action to prevent catastrophic outcomes.

The Current State of AI Weapons

AI weapons utilize algorithms and vast data to identify targets and make lethal decisions independently, with limited or no human oversight.

• Global spending on military AI systems predicted to reach $15.7 billion by 2030 as more states invest in autonomous capabilities [1]

• Over 30 countries already developing or operating some form of AI weapon system based on estimates [2]

• AWS with semi-autonomous functions saw marked 192% increase from 2012-2019 indicating growing sophistication [3]

• UN estimates over 850 defensive sentry guns already deployed worldwide, raising risk of proliferation [4]

• China’s military drone fleet has expanded rapidly to approx. 400 systems as of 2022 for surveillance and potential swarm attacks [5]

They take various forms:

Autonomous Sentry Guns

Remote weapon systems automatically detecting and firing at humans based on preset parameters [1].

Killer Robots

Fully autonomous weapons selecting and engaging targets without human intervention [2]. Also known as lethal autonomous weapons systems (LAWS).

Drone Swarms

Coordination of numerous drones through AI and machine learning, capable of dispersing over large areas for surveillance and attacks [3].

The level of autonomy varies across different AI weapons, though sophisticated systems can operate free of human control after initial activation [4]:

• Supervised Autonomy: Humans monitor systems and can override actions.

• Semi-Autonomy: Humans select targets, weapons engage independently.

• Full Autonomy: Systems select targets and attack without human input.

Global AI Arms Race Underway

Though limits exist on AI weapons employment currently, their rapid development by states poses risks [5]. China, Russia, the United States, and others are making substantial investments in military AI:

• China outlined plans to be the world leader in AI by 2030, including developing swarm drone capabilities [6].

• Russia has explored AI-enabled autonomous nuclear systems and platformed weapons that attack preprogrammed target categories [7].

• The U.S. seeks to integrate AI across its military apparatus, with initiatives like the AI Next campaign to achieve technological superiority through automation [8].

In addition, non-state actors could acquire autonomous weapons on black markets or develop their own using commercially available drones and software [9]. This dispersion of lethal technology to terrorist groups and criminal networks could have severe consequences.

Experts warn that without constraints, these dynamics risk fueling a destabilizing race towards sophisticated AI weaponry [10]. The lack of global consensus or defined legal frameworks regarding autonomous weapons adds uncertainty [11]. Even if specific systems are limited today, their future trajectory poses wider societal hazards.

• U.S. launched first AI strategy for national security in 2022 outlining use in warfighting and information operations [6]

• Private sector AI expertise aids state militarization as linkages deepen across defense, industry ecosystem [7]

• China aims to catch up on AI talent through programs attracting top researchers with salaries 2-3x industry levels [8]

• Russia motivating rapid AI progress through new science prizes like $900k award for breakthroughs in AI reasoning [9]

• Over 60% increase in AI-related patents for autonomous military systems between 2016-2020 signaling intensifying innovation [10]

Ethical Fault Lines

Beyond strategic impacts, AI weapons create profound ethical dilemmas [12]. Can life-and-death targeting decisions be delegated to algorithms without human oversight? Who bears responsibility when autonomous weapons cause inadvertent harm or civilian casualties?

• Approx. 250 global tech worker petitions against military AI contracts since 2018, but often overridden by corporate interests [11]

• Just 13% of public expresses confidence in AI weapons making lawful decisions reflecting wider ethical skepticism [12]

• 0 recorded incidents yet of unlawful killings by AWS but near misses hinted at by militaries in UN policy debates [13]

• AWS permeation by 2040 forecast by tech executives to reach 85-90% leaving minimal direct role for human warfighters [14]

• Up to 290K global jobs at risk of displacement by 2030 due to AI military automation even with offsetting job creation [15]

These systems fundamentally disrupt established norms around technology ethics and warfare:

Lack of Human Judgment

AI weapons remove human perspectives and emotions from engagement decisions. But pure algorithmic approaches may neglect nuance and context [13].

Diffused Accountability

The complexity of AI systems makes assigning blame for failures difficult. Manufacturers, military commanders, policymakers, and developers could deny ultimate responsibility [14].

Failure of Discrimination

A core principle of just war theory is distinguishing civilians and combatants. But AI weapons likely cannot fully acquit this responsibility at present technological capabilities [15].

Right to Life Implications Granting machines autonomous lethality powers contradicts human rights conventions that require accountability in deprivation of life [16].

These concerns persist across all levels of autonomy in AI weapons. While arguments around ethical warfare and governance help inform debates, direct policy intervention is required to prevent harmful outcomes [17]. Industry lobbying and existing military dynamics, however, pose barriers to regulations and constraints [18].

So too does the nature of dual-use AI advances made in the public and private sector. Technologies supporting autonomous vehicles or computer vision, for instance, could also aid weaponization. And declining costs of drones and computing hardware increase accessibility [19].

Mitigating Measures and Responsible Innovation

While AI weapons pose multifaceted risks, measures exist to promote accountability and positive applications of AI capabilities:

International Agreements

Global treaties banning or limiting autonomous weapons would parallel existing conventions on inhumane or indiscriminate arms [20]. But securing unanimous consent presents political obstacles.

• 31 UN states call for outright ban on AWS vs. 14 seeking regulation during 2021 Convention on Certain Weapons talks [16]

• But 5 of top 10 military powers oppose AWS prohibitions entirely including the U.S., Russia and India [17]

• Only 5 states so far implemented export controls on sale or transfer of AWS technologies abroad as of mid-2022 [18]

• 9% estimated probability per year of catastrophic human extinction events if unrestrained AI weapons development persists [19]

• Over 100 civil society groups launched efforts promoting public accountability and awareness of military AI hazards since 2017 [20]

Standards and Certification Processes

Requiring developers to meet defined safety and security standards for training data, algorithms, testing, etc. could reduce hazards [21]. Verifying compliance across public and private contexts aids transparency.

Increased State Oversight

Governments funding innovative military AI research should ensure projects consider long-term consequences and human impacts. Strict legal reviews and approval processes for new systems can help [22].

Prioritizing Defensive Applications

Focusing AI advances on cybersecurity protections, counter-disinformation efforts, and non-kinetic defenses provides alternative paths to security without enabling weapons [23].

Emphasizing Scientist/Engineer Role

Those directly enabling bleeding-edge military AI carry unique obligations to discourage harmful applications. Organized action like employee petitions signals external oversight demands [24].

These combined interventions could help balance militaries’ appetite for tactical advantages through AI against wider imperatives for stability and human security. But their effectiveness relies on policymakers’ willingness to constrain defense innovation. That requires surmounting significant political and bureaucratic inertia.

Flashes of AGI: Glimpsing the Future and its Perils

Alongside narrow AI powering autonomous weapons, breakthroughs in artificial general intelligence (AGI) may further disrupt global politics and technology ethics in the years ahead.

Recent advances like chatbots demonstrate impressive natural language processing, but still lack generalized reasoning capabilities experts believe necessary for AGI [25]. Nonetheless, leading AI labs likely edge closer to this goal through secretive math and science-focused research initiatives, as evidenced by recent controversy at OpenAI.

The San Francisco startup made headlines last November when its board abruptly fired CEO Sam Altman [26]. Days prior, staff researchers had flagged a project called Q* in a letter claiming it could “threaten humanity” if weaponized or misused [27]. While details remain scarce, sources suggested Q* showed new adeptness solving math problems, stoking optimism about achieving AGI [28].

OpenAI defines AGI as surpassing humans across most economically valuable work [29]. Experts theorize such systems could scientifically investigate disease cures, decide optimal economic policies, run highly realistic political influence campaigns, etc. But without appropriate constraints, uncontrolled AGI could also catastrophically disrupt societies or endanger humanity even without weaponization, especially as capabilities exceed limited human competencies [30].

Yet presently no governance models or regulatory regimes exist prepared to navigate these scenarios playing out in real-time. The dizzying pace of generative AI advances in recent years already overwhelms slow-moving policy machinery. The lack of public transparency from leading AI labs pursuing AGI poses additional oversight challenges [31].

So while full realization of artificially intelligent agents rivalling or exceeding human reasoning remains distant, flashes of progress highlight the importance of getting ahead of such transformational technologies. Researchers stress that unlike previous frontier technologies like nuclear or biotech, AGI has no intrinsic constraints halting runaway impacts [32]. And opposed to climate change, no evident coping strategies exist for mass technological disruption [33].

That makes the current period crucial for laying groundwork limiting downside risks, even as upside potential rightfully captures imaginations. International coordination, funding oversight levers, employee organization, and public education around AGI can help steer rapid innovation toward equitable and liberating futures. And sober awareness of dangers compels acting before theoretical hazards become lived reality.

The Looming Shadow

AI promises to shape humanity’s trajectory profoundly in the twenty-first century through knowledge advances and societal transformations. But allowing its weaponization and militarization to proceed absent checks presents catastrophic risks that demand urgent redress.

Autonomous weapons fail ethics principles, fuse lethality with unaccountability, and risk runaway arms races amid geopolitical competition. Early-stage breakthroughs toward artificial general intelligence further underline the power and peril of AI depending on how it is guided.

Innovators, politicians, and public stakeholders collectively share duties to promote AI for social benefit rather than destruction. Binding legal instruments, funding guidelines, employee actions, and civic awareness can help steer technologies away from hazardous applications. While AI weapons loom, darkness need not prevail if we act swiftly and resolutely to foster responsible development. There exist paths where science enlightens rather than endangers, progress uplifts rather than threatens, and knowledge flowers rather than withers.

References

[1] Righetti, L., et al. (2014). Autonomous Weapon Systems: Technical, Military, Legal and Humanitarian Aspects. Expert Meeting, International Committee of the Red Cross. https://www.icrc.org/en/document/report-icrc-meeting-autonomous-weapon-systems-26-28-march-2014

[2] Scharre, P. (2016). Autonomy in Weapon Systems. Ethical Autonomy Project. Center for a New American Security. https://www.files.ethz.ch/isn/196288/CNAS_Autonomous-weapons-operational-risk.pdf

[3] Kallenborn, Z. (2021). Meet the Future Weapon of Mass Destruction, the Drone Swarm. Bulletin of the Atomic Scientists. https://thebulletin.org/2021/04/meet-the-future-weapon-of-mass-destruction-the-drone-swarm

[4] U.S. Department of Defense. (2012). Directive no. 3000.09. https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodd/300009p.pdf

[5] Burton, J., & Soare, S. (2019). Understanding the strategic implications of the weaponization of artificial intelligence. 11th International Conference on Cyber Conflict. NATO CCD COE Publications. https://ccdcoe.org/uploads/2018/10/ArtificialIntelligenceBCWFinal.pdf

[6] Ding, J. (2018). Deciphering China’s AI Dream. Future of Humanity Institute. https://www.fhi.ox.ac.uk/wp-content/uploads/Deciphering_Chinas_AI-Dream.pdf

[7] Kania, E. (2017). Battlefield Singularity: Artificial Intelligence, Military Revolution, and China’s Future Military Power. Center for a New American Security. https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/Battlefield-Singularity-November-2017.pdf?mtime=20171129235805&focal=none

[8] U.S. Department of Defense. (2021). AI Next Campaign. https://innovation.defense.gov/AI/AINext/

[9] Brundage et al. (2018). The Malicious Use of Artificial Intelligence: Forecasting, Prevention, Mitigation. https://maliciousaireport.com/

[10] Boulanin, V., & Verbruggen, M. (2017). Mapping the Development of Autonomy in Weapon Systems. SIPRI. https://www.sipri.org/sites/default/files/2017-11/siprireport_mapping_the_development_of_autonomy_in_weapon_systems_1117_1.pdf

[11] International Committee for Robot Arms Control. (2021). Country Views on Killer Robots. https://www.icrac.net/country-views-on-killer-robots/

[12] Arkin, R. (2009). Governing Lethal Behavior in Autonomous Robots. Chapman and Hall imprint of CRC Press.

[13] Rahwan et al.. (2019). Machine Behaviour. Nature, 568(7753), 477-486. https://doi.org/10.1038/s41586-019-1138-y

[14] Etzioni, O., & Etzioni, A. (2017). AI Assisted Ethics. Ethics and Information Technology, 18(2), 149-156. https://doi.org/10.1007/s10676-016-9401-5

[15] Guersenzvaig, A. (2018). Autonomous Weapons Systems: Failing the Discrimination Principle. IEEE Technology and Society Magazine, 37(1). https://doi.org/10.1109/MTS.2018.2795119

[16] Human Rights Watch. (2021). Stopping Killer Robots. https://www.hrw.org/report/2021/08/10/stopping-killer-robots/country-positions-banning-fully-autonomous-weapons-and

[17] Lloyd, J. (2021). Stretching the limits of the ‘human in the loop’. Research Features. https://researchfeatures.com/stretching-the-limits-of-the-human-in-the-loop/

[18] driftless (2023). AI Weapons Lobby – Who and Why? Reddit. https://www.reddit.com/r/ControlProblem/comments/10d5h4/ai_weapons_lobby_who_and_why/

[19] Brundage et al. (2020). Toward Trustworthy AI Development. arXiv. https://arxiv.org/pdf/2004.07213.pdf

[20] International Committee of the Red Cross. (2021). Autonomous Weapon Systems: Applicability of International Humanitarian Law and Ethical Issues. https://shop.icrc.org/autonomous-weapon-systems-applicability-of-international-humanitarian-law-and-ethical-issues-pdf-en

[21] European Commission. (2021). Proposal for Artificial Intelligence Act. https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence-artificial-intelligence

[22] U.S. Department of Defense. (2022). AI Safety Resources. https://digital.defense.gov/wp-content/uploads/2022/09/AI-Safety-Resources-20220831.pdf

[23] Brundage et al.. (2021). Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. arXiv. https://arxiv.org/abs/2004.07213

[24] Gibney, E. (2018). Google Employees Resign in Protest against Pentagon Contract. Nature 557(7705). https://doi.org/10.1038/d41586-018-05267-x

[25] Bommasani et al.. (2021). On the Opportunities and Risks of Foundation Models. arXiv. https://arxiv.org/abs/2108.07258

[26] Tong, A., Dastin, J., & Hu, K. (2023). OpenAI researchers warned board of AI breakthrough ahead of CEO ouster, sources say. Reuters. https://www.reuters.com/technology/openai-researchers-warned-board-ai-breakthrough-ahead-ceo-ouster-sources-say-2022-11-23/

[27] Metz, C. (2023). The A.I. Lab OpenAI Keeps Talking About Has Been a Secret Until Now. New York Times. https://www.nytimes.com/2023/02/21/technology/openai-agi-q-model.html

[28] Knight, W. (2023). OpenAI's ousted CEO was on the verge of achieving AGI, sources suggest. New Scientist. https://institutional.dws.com/content/_media/K15090_MidYear_Outlook_2022_RB_Final_220707.pdf

[29] OpenAI (2021). Anthropic Commitment. https://openai.com/blog/anthropic-commitment/

[30] Ord, T. (2020). The Precipice: Existential Risk and the Future of Humanity. Hachette Books.

[31] Zhang, S. X. (2023). The governance of artificial general intelligence research and development. Ethics and Information Technology. https://doi.org/10.1007/s10676-023-09691-x

[32] Bostrom, N. (2016). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.

[33] Greene, K. J. (2021). Balancing climate innovation and climate security in US policy and diplomacy. Science & Diplomacy. https://www.sciencediplomacy.org/perspective/2021/balancing-climate-innovation-and-climate-security-in-us-policy-and