Autonomous machines are increasingly taking over human tasks. Initially, simple and limited assignments such as assembly line labor were taken over by machines. Nowadays, more complex tasks are transferred to sofware and robots. Even parts of jobs that were previously deemed purely human occupations, such as being a driver, credit line assessor, medical doctor, or soldier are progressively carried out by machines (e.g., [36, 42]). As many believe, ceding control over important decisions to machines requires that they act in morally appropriate ways. Or, as Picard puts it, “the greater the freedom of a machine, the more it will need moral standards” [102, p. 134].
For this reason, there has been a growing interest in Machine Ethics, defned as the discipline “concerned with the consequences of machine behavior towards human users and other machines”[6, p. 1].1 Research in this feld is a combination of computer science and moral philosophy. As a result, publications range from theoretical essays on what a machine can or should do (e.g. [46, 122]), to prototypes implementing ethics in a system (e.g., [3, 142]). In this feld, the emphasis lies on how a machine could act ethically in an autonomous fashion.
Tere have been several atempts to classify current approaches of machine ethics. A frst high-level classifcation was proposed by Allen et al. [2] in 2005, distinguishing between top-down theory-driven approaches, botom-up learning approaches, and hybrids of the two. Subsequent work tried to further determine types of procedures [29, 65, 147], but these works were either mixing diferent dimensions (e.g., mixing technical approach and ethical theory in one category) [147] or ofering an orthogonal dimension that did not ft the existing taxonomy (e.g., whether normative premises can difer between ethical machines) [29]. Also, because these works did not provide an extensive and systematic overview of the application of their taxonomy, verifcation of the taxonomy with papers from the feld was missing. A recent survey from Yu et al. [65] on ethics in AI has some overlap with this work, but 1) does not systematically apply the ethical theory classifcation to selected papers and 2) takes a broader perspective to include consequences of and interaction with ethical AI, while this paper focuses specifcally on machine ethics implementations. Hence, compared to previous works, this survey covers more related work, provides a more extensive classifcation, and describes the relationship between diferent ethics approaches and diferent technology solutions in more depth than previous work [2, 29, 65, 147]. Furthermore, gaps are identifed in the implementation of ethics in existing systems.
Tis paper is created as a collaboration between ethicists and computer scientists. In the context of implementing machine ethics, it can be a pitfall for philosophers to use a purely theoretical approach without consulting computer scientists, as this can result in theories that are too abstract to be implemented. Conversely, computer scientists may implement a faulty interpretation of an ethical theory if they do not consult a philosopher. In such an interdisciplinary feld, it is crucial to have a balanced cooperation between the diferent felds involved.
Te contributions of this article are as follows:
• Based on previous work [2], a taxonomy is defned to analyze the feld based on three diferent dimensions: types of ethical theory (Section 4), implementation of ethics (Section 5), and technological details (Section 6).
• An exhaustive selection and description of relevant contributions related to machine ethics implementations is presented (Section 8).
• Te reviewed publications are classifed and research paterns and challenges are identifed (Section 7).
• A number of general lessons for the feld are discussed and further important research directions for machine ethics are outlined (Section 9).
As such, this survey aims to provide a guide, not only to researchers but also to those interested in the state of the art in machine ethics, as well as seed a discussion on what is preferred and accepted in society, and how machine ethics should be implemented.
Te rest of this paper is structured as follows. Section 2 introduces the feld of machine ethics, its importance, and justifcation of used terminology throughout the paper. Section 3 lists the methodology used to create this survey, including the search methodology, the process of creating the classifcation dimensions, and the actual classifcation process. Section 4, 5 and 6 introduce the three classifcation dimensions presented in this survey. Section 7 discusses the results of the classifcation of the selected papers, whereas Section 8 summarizes those papers. Finally, Section 9 outlines which future avenues of research may be interesting to pursue based on the analysis, as well as the limitations of this survey.
Before going into more depth on the implementation of ethics, it is important to establish what is considered machines ethics, why it maters, and present the relevant terminology for the feld.
2.1 Relevance
Sofware and hardware (combined under the term “machine” throughout this survey) are increasingly assisting humans in various domains. Tey are also tasked with many types of decisions and activities previously performed by humans. Hence, there will be a tighter interaction between humans and machines, leading to the risk of less meaningful human control and an increased number of decision made by machines. As such, ethics needs to be a factor in decision making to consider fundamental problems such as the atribution of responsibility (e.g., [122]) or what counts as morally right or wrong in the frst place (e.g., [135]). Additionally, ethics is needed to reduce the chance of negative results for humans and/or to mitigate the negative efects machines can cause.
Authors in the feld give diferent reasons for studying (implementations in) machine ethics. Fears of the negative consequences of AI motivate the frst category of reasons: creating machines that do not have a negative societal impact [12, 86]. With further autonomy and complexity, ethics need to be implemented in a more elaborate way [24, 38, 49, 75, 95, 99]. Society needs to be able to rely on machines to act ethically when they gain autonomy [6, 48]. A second category of reasons for studying machine ethics focuses on the ethics part: by implementing ethics, ethical theory will be beter understood [24, 58, 95, 99].Robots might even outperform humans in terms of ethical behavior at some point [4, 8].
Some authors contend that in cases with no consensus on the most ethical way to act, the machine should not be allowed to act autonomously [5, 122]. However, not acting does not imply the moral conundrum is avoided. In fact, the decision not to act also has a moral dimension [55, 78, 144] —think, for example, of the diference between active and passive euthanasia [106]. Additionally, by not allowing the machine to act, all the possible advantages of these machines are foregone. Take, for example, autonomous cars: a large number of trafc accidents could be avoided by allowing autonomous cars on the road. Moreover, simply not allowing certain machines would not stimulate the conversation on how to solve the lack of consensus; a conversation that can lead to new, more practical ethical insights and helpful machines.
2.2 Terminology
An ofen-used term in the feld of machine ethics is “Artifcial Moral Agent” or AMA, to refer to a machine with ethics as part of its programming. However, to see if this term is appropriate to use, it is important to identify what ethical agents mean in the context of machine ethics and how ethical machines should be regarded. In an ofen-cited paper, Moor [95] defnes four diferent levels of ethical agents:
Ethical-impact agents are types of agents that have an (indirect) ethical impact. An example would be a simple assembly line robot that replaces a human in a task. Te robot itself does not do anything (un)ethical by acting. However, by existing and doing its task, it has an ethical impact on its environment; in this case, the human that performed the task is replaced and has to fnd another job.
Implicit ethical agents do not have any ethics explicitly added in their sofware. Tey are considered implicitly ethical because their design involves safety or critical reliability concerns. For example, autopilots in airplanes should let passengers arrive safely and on time.
Explicit ethical agents draw on ethical knowledge or reasoning that they use in their decision process. Tey are explicitly ethical, since normative premises can be found directly in their programming or reasoning process.
Fully ethical agents can make explicit judgments and are able to justify these judgments. Currently, humans are the only agents considered to be full ethical agents, partially because they have consciousness, free will, and intentionality.
While these defnitions can help with a frst indication of the types of ethical machines, they do not allow for distinctions from a technical perspective and are also unclear from a philosophical perspective: Moor [95] does not actually defne what an (ethical) agent is. For example, it can be debated whether an autopilot is an agent. Terefore, a clearer defnition is needed of what an agent is. Himma [74] investigates the concepts of agency and moral agency, drawing from philosophical sources such as the Stanford Encyclopedia of Philosophy and Routledge Encyclopedia of Philosophy. He proposes the following defnitions:
Agent : “X is an agent if and only if X can instantiate intentional mental states capable of performing actions.” [74, p. 21]
Moral agency : “For all X, X is a moral agent if and only if X is (1) an agent having the capacities for (2) making free choices, (3) deliberating about what one ought to do, and (4) understanding and applying moral rules correctly in the paradigm cases.” [74, p. 24]
With regards to artifcial agents, Himma postulates that the existence of natural agents can be explained by biological analysis, while artifcial agents are created by “intentional agents out of pre-existing materials” [74, p. 24]. He emphasizes that natural and artifcial agents are not mutually exclusive (e.g. a clone of a living being). He further claims that moral agents need to have conscious and intentionality, something that state-of-the-art systems do not seem to instantiate. It is worth noting that Himma atempts to provide a general defnition of moral agency, while for example Floridi and Sanders [54] propose to change the current description of a moral agent. For example, they proposed description includes the separation the technical concepts of moral responsibility and moral accountability, a distinction that was not evident thus far: “An agent is morally accountable for x if the agent is the source of x and x is morally qualifable […] To be also morally responsible for x, the agent needs to show the right intentional states”. Wallach and Allen [137] rate AMAs along two dimensions: how sensitive systems are to moral considerations and how autonomous they are. Sullins [126] has a partially overlapping concept of requirements for robotic moral agency with Himma’s, that intersects with Wallach and Allen’s relevant concepts: autonomy (i.e. “the capacity for self-government”) [27]), intentionality (i.e. “the directedness or ‘aboutness’ of many, if not all, conscious states”[27]) and responsibility (i.e. “those things for which people are accountable”[27]).
Tese are just some notions of how concepts such as agency, autonomy, intentionality, accountability and responsibility are important to the feld of machine ethics. However, it is challenging to summarize and defne these concepts concisely while doing justice to the work in philosophy, and computer science that has been done so far, including the discussions and controversy around diferent relevant concepts (as the diferent concepts of moral agency display). Te goal of this survey is not to give an introduction to moral philosophy, so this section merely gives a glimpse of the depths of the topic. Rather, the goal is to summarize and analyze the current state of the feld of machine ethics. To avoid any assumption on concepts, the popular term Autonomous Moral Agent is not used in this survey: as shown above, the term “agent” can be debated in this context and the term “autonomous” has various meanings in the diferent surveyed systems. Instead, a machine that has some form of ethical theory implemented—implicitly or explicitly—in it is referred to as an “ethical machine” throughout this paper.
Tis section describes the search strategy, paper selection criteria, and review process used for this survey.
3.1 Search Strategy
A literature review was conducted to create an overview of the diferent implementations of and approaches to machine ethics. Te search of relevant papers was conducted in two phases: automated search and manual search.
Automated Search. Te frst phase used a search entry that refected diferent terms related to machine ethics combined with the word ‘implementation’:
implementation AND ( ”machine ethics” OR ”artifcial morality” OR ”machine morality” OR ”computational ethics” OR ”roboethics” OR ”robot ethics” OR ”artifcial moral agents”) Tese terms were cumulated during the search process (e.g. [139, p. 455]); each added term resulted in a new search until no new terms emerged.2 No time period of publication was specifed, to include as many items as possible.
Te following library databases were consulted (with the number of results in parenthesis): Web of Science (18), Scopus (237), ACM Digital Library (16), Wiley Online Library (23), ScienceDirect (48), AAAI Publications (4), Springer Link (247), and IEEE Xplore (113). Of these initial results, 37 items were selected based on the selection criteria listed in Section 3.2.
Manual Search. Te second phase included checking the related work and other work by the same frst authors of phase one. Twenty-nine promising results from phase one did not meet all criteria, but were included in the second search phase to see if related publications did meet all criteria. Tis process was repeated for each newly found paper, until no more papers could be added that ft the selection criteria (see 3.2). Tis resulted in a total of 49 papers, describing 48 ethical machines.
3.2 Selection Criteria
Afer the selection process, two more coauthors judged which papers should be in- or excluded to verify the selection. Papers were included only if they adhered to all of the following inclusion criteria. Te paper
• implements a system OR describes a system in sufcient (high-level) detail for implementation OR implements/describes a language to implement ethical cases,
• describes a system that is explicitly sensitive to ethical variables (as described by [95]), no mater whether it achieves this sensitivity through top-down rule-based approaches or botom-up data-driven approaches (as described by [2]),
• is published as a conference paper, workshop paper, journal article, book chapter, or technical report,
• and has ethical behavior as the main focus.
Te following exclusion criteria were used. Te paper
• describes machine ethics in a purely theoretical fashion, • describes a model of (human) moral decision making without an implementable model description,
• lists results of human judgment on ethical decisions without using the data in an implementation,
• is published as a complete book, presentation slides, editorial, thesis, or has not been published,
• describes a particular system in less detail than other available publications, • focuses on unethical behavior to explore ethics (e.g., a lying program), • mentions ethical considerations while implementing a machine, but does not focus on the ethical component and does not explain it in enough detail to be the main focus,
• simulates artifcial agents to see how ethics emerge (e.g. by using an evolutionary algorithm without any validation),
• and describes a general proposal of an ethical machine without mentioning implementation related details.
Given the focus on algorithms implementing moral decision making and the limitations of space, we will not go into further detail as regards recent interesting work on AI and moral psychology (cf. e.g. [17, 28, 85, 118]).
3.3 Taxonomy Creation and Review Process
In order to be able to identify strengths and weaknesses of the state of the art, we created diferent taxonomies and classifed the selected papers accordingly. It was clear that both an ethical taxonomy (cf. Table 1) and a technological taxonomy (cf. Table 4) were necessary. All authors agreed that there were some aspects of implementations that did not relate exclusively to either the ethical or the technical dimension but were still important for the classifcation. Hence, we defned and applied a third taxonomy (cf. Table 3) related to implementation choices. Te frst version of these three taxonomies was created using knowledge obtained during the paper selection.
Before the classifcation process started, one third of the papers were randomly selected to review the applicability of the taxonomy proposed and adjust the assessment scheme where required. Any parts of the taxonomies that was unclear and led to inconsistent classifcations was adapted and clarifed.
Each selected paper was categorized according to the features of the three diferent taxonomies
(discussed in Sections 4–6). Since the ethical classifcation is perhaps the most disputable, it was determined by three distinct assessors: two philosophers and a computer scientist. Two computer scientists evaluated the implementation and technical details of all proposed systems.
To provide a classifcation for all selected papers, multiple classifcation rounds took place for each dimension. In between classifcation rounds, disagreements across reviewers were discussed until a consensus was reached. In the case of the ethical dimensions, four papers could not be agreed upon afer multiple classifcation rounds. As such, these papers were labeled as ’Ambiguous’.
Tis section introduces the frst of three taxonomies introduced in this paper: a taxonomy of types of ethical theories, which is the basis for the categorization of ethical frameworks used by machines (in Section 7). Note that this section is not a general introduction to (meta-)ethics, which can for example be found in [26, 41, 94].
4.1 Overview of Ethical Theory Types
It is commonplace to diferentiate between three distinct overarching approaches to ethics: consequentialism, deontological ethics, and virtue ethics. Consequentialists defne an action as morally good if it maximizes well-being or utility. Deontologists defne an action as morally good if it is in line with certain applicable moral rules or duties. Virtue ethicists defne an action as morally good if, in acting in a particular way, the agent manifests moral virtues. Consider an example: an elderly gentleman is harassed by a group of cocky teenagers on the subway and a resolute woman comes to his aid. Te consequentialist will explain her action as good since the woman maximized the overall well-being of all parties involved—the elderly gentleman is spared pain and humiliation which outweighs the teenagers’ amusement. Te deontologist will consider her action commendable as it is in accordance with the rule (or duty) to help those in distress. Te virtue ethicist, instead, will deem her action morally appropriate since it instantiates the virtues of benevolence and courage.
Consequentialist theories can be divided into two main schools: according to act utilitarianism, the principle of utility (maximize overall well-being) must be applied to each individual act. Rule utilitarians, by contrast, advocate the adoption of those and only those moral rules that will maximize well-being. Cases can thus arise where an individual action does not itself maximize well-being, yet is consistent with an overarching well-being maximizing rule. While act utilitarians would consider this action morally bad, rule utilitarians would consider it good.
Deontological ethics can be divided into agent-centered and patient-centered approaches. Agentcentred theories focus on agent-relative duties, such as, for instance, the kinds of duties someone has towards their parents (rather than parents in general). Teories of this sort contrast with patient-centered theories that focus on the rights of patients (or potential victims), such as the right, postulated by Kant, not to be used as a means to an end by someone else [81].
Finally, there are some approaches that question the universal applicability of general ethical principles to all situations, as put forward by deontological ethics, virtue ethics or consequentialism. For such a particularist view, moral rules or maxims are simply vague rules of thumb, which cannot do justice to the complexity of the myriad of real-life situations in which moral agents might fnd themselves. Hence, they have to be evaluated on a case-by-case basis.
We would like to highlight that a moral theory is a set of substantial moral principles that determine what, according to the theory, is morally right and wrong. Moral theories can take diferent structures —they might state their concrete demands in terms of hard rules (deontological ethics); virtues that should guide actions, with reference to an overall principle of utility maximization, or else reject the proposal that there is a one-size-fts-all solution (itself a structural trait, this would be particularism). In this work, we are interested in these structures, which we label “ethical theory types”.
4.2 Categorizing Ethical Machines by Ethical Theory Type
Based on the distinct types of ethical theories introduced above, this sub-section develops a simple taxonomy for categorizing ethical machines, summarized in Table 1.
Table 1. Ethical theory types taxonomy An evaluation of existing approaches to moral decision making
in machines can make use of this typology in the following way. Deontological ethics is rule-based. What maters is that the agent acts in accordance with established moral rules and/or does not violate the rights of others (whose protection is codifed by specifed rules). Accidents occur, and a well-disposed agent might nonetheless bring about a harmful outcome. On of-the-shelf deontological views, bad outcomes (if non-negligently, or at least unintentionally, brought about) play no role in moral evaluation, whereas the agent’s mental states (their intentions, , and beliefs) are important. If John, intending to deceive Sally about the shortest way to work, tells the truth (perhaps because he himself is poorly informed), a Kantian will consider his action morally wrong, despite its positive consequence.3 In the context of machine ethics, the focus is solely on agent relative duties. Hence, no distinction is made between agent-centered and patient-centered theories of deontological ethics in the taxonomy summarized in Table 1. Consequentialists, by contrast, largely disregard the agent’s mental states and focus principally on outcomes: what maters is the maximization of overall well-being. Note that, procedurally, a rule-utilitarian system can appear very similar to a deontological one. Te agent must act in keeping with a set of rules (potentially the very same as in a Kantian system) which, in the long run, maximizes well-being. However, the two types of systems can still be distinguished in terms of the ultimate source of normativity (well-being vs. good will) and will—standardly—difer in terms of the importance accorded to the agent’s mental states. Tus far, nearly all consequentialist machine ethics implementations utilize act utilitarianism. For this reason, the distinction between act and rule utilitarianism is not relevant enough to be included in this survey.
Virtue ethics difers from the aforementioned systems in so far as it does not focus principally on (the consequences or rule-consistency of) actions but on agents, and more particularly on whether
Table 2. High-level overview to ethics categories in the context of ethical machine implementation
they exhibit good moral character or virtuous dispositions. A good action is one that is consistent with the kinds of moral dispositions a virtuous person would have.
In contrast to the other three major approaches, on the particularist view, there is no unique source of normative value, nor is there a single, universally applicable procedure for moral assessment. Rules or precedents can guide our evaluative practices. However, they are deemed too crude to do justice to many individual situations. Tus, according to particularism, whether a certain feature is morally relevant or not in a new situation —and if so, what exact role it is playing there— will be sensitive to other features of the situation.
Table 2 gives a schematic overview of key characteristics of the diferent types of ethical systems that might be implemented in an ethical machine. Note that it does not take some of the more fne-grained aspects diferentiating the theories (e.g., the before-mentioned complications regarding act and rule utilitarianism) into account.
As an alternative to implementing a single determinate type of ethics, systems can also combine two or more types, resulting in a hybrid ethical machine. Tis approach seems enticing when one theory alleviates problems another one might have in certain situations, but it can also generate conficts across types of ethical approaches. Hence, some proposals enforce a specifed hierarchy, which means that one theory is dominant over the other(s) in the system. For example, a primarily deontological system might use rules, but turn to the utilitarian approach of maximizing utility when the rules are in confict. In other cases, the hierarchy is non-specifc and diferent theories are present without a specifed dominant theory.
Some authors do not setle on a particular type of ethical theory. Instead, they provide a confgurable technical framework or language and exhibit how diferent types of ethical theories can be implemented. Te choice of which theory type should be selected is essentially lef to the person implementing the system in an actual use case.
Finally, some contributions were classifed as ambiguous from a meta-ethical perspective. For these, not enough details were given by the authors to classify a paper, or the theories used to implement were not ethical theories but retrieved from domains other than moral philosophy.
Te next section introduces the second dimension of ethical machines: implementing ethics into a machine.
Te second taxonomy that was created for this survey considers the implementation aspect of machine ethics. An important part of creating an ethical system is to decide how to implement ethics. Tat entails defning whether an implementation can follow diferent approaches, how to evaluate the system, and whether or not domain specifcations need to be taken into account. Important features concerning the implementation dimension are summarized in Table 3. Furthermore, this section highlights the implementation challenges that the various ethical theories entail.
5.1 Approaches
Diferent taxonomies have been proposed to determine how ethics types are implemented. Te most infuential and widely referenced scheme, also applied in this survey, stems from Allen, Smit and Wallach [2]. Tey distinguish three types of implementation approaches, namely top-down, botom-up, and hybrid.
Top-down approaches : Top-down approaches assume that humans have gathered sufcient knowledge on a specifc topic; it is a mater of translating this knowledge into an implementation. Te ethical theory types described in Section 4 are examples of normative human knowledge that can be translated into usable mechanisms for machines. Te system acts in line with predetermined guidelines and its behavior is therefore predictable. In AI, strategies using a top-down approach mostly make use of logical or case-based reasoning. Given general domain knowledge, the system can reason about the situation that is given as input. Usually, human knowledge is not specifed in a very structured or detailed way for concrete cases, so knowledge needs to be interpreted before it can be used. Tis process presents the risk of losing or misrepresenting information. Te positive aspect of this approach is that existing knowledge is applied and no new knowledge needs to be generated.
Bottom-up approaches : A diferent method to implementing ethics is to assume the machine can learn how to act if it receives as input enough correctly labeled data to learn from. Tis approach, not just in machine ethics but in general, has gained popularity afer the surge of machine learning in AI and the recent success of neural networks. Technologies such as artifcial neural networks, reinforcement learning, and evolutionary computing fall under this trend. Increased computing power and amounts of data allow learning systems to become more successful. However, data has to be labeled consistently and the right data properties need to be described in a machine-processable way to obtain an accurate training of machines. Tere is a risk that the machine learns the wrong rules or cannot reliably extrapolate to cases that were not refected in its training data. However, for certain tasks, such as feature selection or classifcation, machine learning can be very successful.
Hybrid approaches : As the term suggests, hybrid approaches combine top-down and botom-up approaches. As Allen et al. phrase it: “Both top-down and botom-up approaches embody diferent aspects of what we commonly consider a sophisticated moral sensibility.” [2, p 153] Tey indicate that a hybrid approach is considered necessary, if a single approach does not cover all requirements of machine ethics. Te challenge consists in appropriately combining features of top-down and botom-up approaches.
Bonnemains et al. [29] suggest adding a fourth category, called “Personal values/ethics system”. Essentially, it acknowledges that two diferent agents may rely on diferent ethical systems or may rely on diferent precedence in case of conficts in a hybrid system. In this survey, this is regarded as diversity consideration: the authors of a machine ethics paper consider the possibility that not all ethical machines adhere to the same ethical theory type, and their contribution includes the choice of diverse types of ethics to be implemented. As Bonnemains et al. recognize, this category is somewhat orthogonal to the previous three, as all of those can be seen to implement distinct normative principles. For example, a machine ethics implementation with diversity consideration could allow for multiple ethical theory types to be implemented (i.e., a top-down approach) or allow for diferent machines to learn diferent types of ethics (i.e., a botom-up approach). It is considered part of the implementation dimension rather than the ethics dimension since diversity considerations can also exist within the same ethical theory, for example, by allowing deontological machines to have diferent rules to adhere to while still all being deontological in nature. Tis survey regards structures of normative frameworks and their implementation rather than substantial normative principles (c.f. Table 3).
5.2 Type of Contribution
Ethical systems can be intended to enact diferent aspects of ethical behavior. Tis section discusses the diferent types of contributions published to implement ethical machines.
Model representation : Tis contribution type focuses on representing current ethical knowledge. Te goal is to determine how to appropriately represent a theory, dilemma, or expert-generated guidelines whilst staying true to the original theory.
Model selection : Given a set of alternative options to implement an ethical machine, some systems limit their action to selecting the most fting elements to be included in the system.
Judgment provision : Tese contributions focus on judging an action given a scenario and a set of possible actions. Example outputs are binary (acceptable/non-acceptable) or responses on a scale (e.g., very ethical to very unethical).
Action selection/execution : Here the proposed system chooses which action is best given multiple possible actions for a scenario. Some systems then assign the action to a human, while others carry out the selected action themselves. Part of the action selection task can also be action restriction, when some possible actions are not morally acceptable (enough).
5.3 Evaluation
Most artifacts—simple or complex, concrete or abstract—can be evaluated in virtue of their capacity to fulfll their constitutive function or purpose. A good knife cuts well, a good thermostat reliably activates the heating if the temperature drops below a predetermined threshold, and a good translation system adequately and idiomatically converts grammatical sentences from one language into another. Whereas there are objective and measurable criteria for the evaluation of thermostats, things are more cumbersome when it comes to moral machines. Tis is not because their purpose does not standardly consist in simply “acting morally”, but in executing certain tasks (taking care of the elderly, counselling suicidal people, evaluating risk of recidivism etc.) in a moral fashion. Much rather, the complication arises from the question of what exactly is to count as executing the task at hand in morally appropriate ways, or against what exactly the behavior of the system should be evaluated.
Tere are objective facts as to whether an image represents a certain type of animal or not. Tese facts constrain whether the image is correctly classifed as representing an animal. Te existence of objective, universal moral values, by contrast, is controversial (cf. e.g., [72, 88, 105, 133]). Furthermore, and as objectivists readily acknowledge, delineating what is morally permissible poses an epistemic challenge of a diferent order than identifying, say, a girafe in an image, or determining the weight of an object. Te ontological and epistemic complications that arise in the moral domain thus make it difcult to setle on standards against which the performance of a moral machine could be evaluated. More fundamentally, it is not even evident what kinds of considerations should guide the process of choosing such standards.
While complications as to the evaluation of a moral machine are worrying, their practical signifcance should not be exaggerated. Although there is disagreement as regards complex cases, in ordinary life situations in which one is confronted with extremely difcult ethical decisions or run away trolleys are exceedingly rare. In many domains, moral dilemmas are unlikely to arise or be of much import, and there is widespread convergence (not only among the folk, but experts, too) on what constitutes adequate moral behavior. Overall, then, the challenge of evaluation might raise metaphysical and epistemic complications of limited pragmatic importance, at least when care is exercised to limit the decision capacity of moral machines to mundane contexts that steer clear of complex ethical paradoxes.
5.3.1 Test. When a system is tested, the system outcome needs to be compared against a ground truth. Tese may have the following origins:
Non-experts : One possibility consists in making folk morality the benchmark. Problematically, there is substantial evidence of moral parochialism across cultures (e.g., [53, 87, 113]), and it is not difcult to fnd topics on which a single nation is roughly divided —just think of abortion, euthanasia, or same-sex relations in the US [112]. Furthermore, the existence of widespread convergence in moral opinion does not necessarily make such opinions true or acceptable (consider that until a century and a half ago, there was broad agreement in considerable parts of the world that slavery is morally acceptable).
Experts : To escape the tyranny of a potentially mistaken or self-serving majority, one might adopt the standard of experts in normative ethics. Problematically, however, experts themselves are sometimes deeply divided on fundamental issues of moral import as well as meta-ethical intuitions [31] and their very expertise can be called into question [119, 120].
Laws : One might side-step the complications raised by retreating to a second-best solution: the law. Tis strategy, however, is not without drawbacks either, as the law is simply silent on most questions of day-to-day morality. It is, for instance, not illegal to lie in most contexts, yet it would be regarded as outrageous to be perpetually deceived by “moral” machines. Still, it might be suitable to draw on the law to provide restrictions where they exist, for example, as concerns the “Laws of War” or “Laws of Armed Conficts” for the lethal weapons domain [8], or specifc domain rules such as the Code of Ethics for Engineers [91]. As Arkin [8] suggests, scoping the problem using domain-specifc requirements can make it more easily implementable and testable.
5.3.2 Prove. Another approach, typically based on some type of logic, consists of proving that the system behaves correctly according to some known specifcations. Tis approach can be divided into the following types:
Model checker : Given an ethical machine, a model checker exhaustively and automatically ascertains that it adheres to a given set of specifcations.
Logical proof : Tis approach provides a logical proof that given certain premises, the system does what it should do. Proofs of this sort can be efected manually, or by using a theorem prover which employs automated logical and mathematical reasoning.
Note that this approach assumes that a correct specifcation exists a priori and is widely accepted. Within the logic community, model checking and theorem proving are ofen considered an implementation issue rather than a type of evaluation (e.g., see [68]). In some cases, authors do not even explicitly mention that they employ a model checker, because it is inherent in their approach to logic programming. However, given the multidisciplinary nature of the feld of machine ethics, it is vital to explicitly state which approach has been used. Furthermore, while logical/internal validity and consistency may be inherent in the system, a form of evaluation is necessary to ensure the system acts as expected in diferent cases and exhibits external validity.
5.3.3 Informal evaluation. Some authors refrain from formally evaluating their implementation. Instead, they only describe their work and, in some cases, show a few example scenarios or exhibit application domains. Whilst these approaches may have limited validity, they may be warranted given the evaluation complications outlined above or when the authors principally engage in theory building [50].
Example scenarios/case studies : To showcase that the system works as intended, one or multiple scenarios are presented to demonstrate the system’s performance. Tis procedure gives a frst indication of the functionalities of the machine or may help in theorizing about certain properties of a system, but it does not cover all possible situations or give a complete performance indication.
Face validity : Ofen described as “reasonable results”, authors using this approach state that the results of a few example tasks are as expected. It is ofen unclear what this means and to what extent these results are desirable.
5.3.4 None. When no evaluation could be discerned, papers were categorized as having none of the evaluation types present.
5.4 Domain Specificity
What is deemed an appropriate action can depend on the domain in which the ethical agent is operating, such as the principles in the domain of biomedical ethics as proposed by Beauchamps and Childress [21] for the medical domain, or the Rules of Engagement and Laws of Armed Confict for autonomous weapon systems [8]. Hence, some contributions focus on a specifc application domain, which limits the scope of an ethical machine implementation, and thus the endeavor is more manageable [8].
Table 3. Ethical implementation taxonomy
5.5 Implementation Challenges
Tere are certain challenges inherent in the diferent types of ethics when they are being implemented. Since these obstacles need to be taken into account to select an ethical theory type for an ethical machine, this subsection provides a (non-exhaustive) list of complications.
Challenges when implementing deontological ethics: At a frst glance, the rule-based nature of deontological ethics seems to lend itself well for implementation. However, at diferent stages of implementation, challenges arise. Te frst issue is which rules should be implemented. Rules are expected to be strictly followed, implying that for every exception, the rule must be amended, resulting in an extremely long rule. Determining the right level of detail is important for the success of an application: when the rules are not practical and at the right level of detail, they will not be interpretable for the machine [8]. Second, there might be conficts between rules [29]—in general or in specifc situations. Whilst ordering or weighing the rules might address this issue from an implementational perspective, determining an order of importance can be difcult. Also, this assumes that all relevant rules are determined before they are used.
Challenges when implementing consequentialist ethics: Tere are three main categories of difculties for consequentialist ethics. First, it is hard to identify consequences and determine the right level of detail and aggregation in terms of time and size. Some outcomes might have resulted regardless of the action theorized to have caused it. In real-life situations, all possible consequences are not always that clear beforehand given the lack of epistemic transparency and causal interdependence.
A second issue is concerned with quantifying consequences. As consequentialism is about maximizing utility, the problem is how to defne utility. In simple scenarios like the Trolley problem, utility is ofen defned as how many people survive or die. In the real world, more complex concepts, such as happiness and well-being, are preferred to defne utility. Tere are measures available (e.g., QALY [64]), but using a diferent measure can give a diferent outcome. Even more so, even if each consequence is assigned a utility, it might still be inappropriate to simply aggregate them (e.g., see [83]).
Finally, there might be a signifcant computational cost when computing utility [141] requiring heuristics or approximations to derive a correct answer in time. Tis, in turn, requires a verifcation of whether these results are still correct.
Challenges when implementing virtue ethics: Virtues are positive character traits, character traits that should be manifested in morally good actions. Defning what “character” a machine has is troubling, if a machine can be claimed to have a character at all. To judge whether a machine—or a human for that mater—is virtuous is not possible by merely observing one action or a series of actions that seem to imply that virtue; the reasons behind them need to be clear [123]. Perhaps the best way to create a virtuous machine is to let a machine mimic the behavior of a virtuous person. But how is a certain virtue measured, and who decides which virtues are more important and how to pick the perfect role model? Coleman [39] even proposes diferent virtues that are more desirable for machines rather than human virtues, implying merely mimicking a virtuous person is not sufcient.
To circumvent these challenges, machine ethics researchers have not used virtue ethics ofen, as the alternatives might be more appealing. For example, Haber [66] states that virtue ethics and principle-based ethics are complements and that for each trait there will be a principle that expresses that trait and vice versa. While not everyone agrees with Haber, it is easier and more detailed from a computational perspective to implement rules than generic virtues to adhere to. Arkin [8] also concludes that principle-based and act-centric models allow for stricter ethical implementations, which is desirable in machine ethics.
Challenges when implementing particularism: In particularism, the system needs to take the entire context into account. Tis implies that it needs to either be trained for all possible cases, which is not possible, or be able to extrapolate without using generalizations, which is highly challenging. For each feature of the context, the system would have to recognize whether it is morally relevant in the given case and how it will infuence the result. Case-based methods or instance-based classifcations come closest to allowing an implementation of particularism. More recently, some contributions are trying to approximate particularist ethics using neural networks (e.g., [62, 71]).
Challenges when implementing hybrid approaches: Each type of ethical theory raises its own set of complications, but combining them introduces additional issues. First, when diferent types of ethical theories are used in a non-hierarchical way, the interaction between them can be problematic: how should the results from diferent ethical approaches be combined to guarantee morally appropriate outcomes? What happens when the results of diferent implemented ethical theory types stand in confict, and how should such conficts be resolved?
Second, when a hierarchical approach is employed, it is not evident when the system should employ one theory rather than another. One standard approach resorts to the secondary set of ethical principles when the frst does not deliver a verdict. While this alleviates some of the challenges of hybrid systems, it is still possible that the second ethical theory proposes something that conficts with the frst ethical theory type.
Te third and fnal taxonomy introduced regards the technical dimension of machine ethics. Tis includes the type of technology chosen for the implementation, the input the system relies on, the ethical machine’s availability (i.e., implementation details are published) and some implementation features: whether it relies on specifc hardware or feedback from users, provides explanations for its conclusions, has a user interface (UI), and whether the input for the system needs to be preprocessed. Important features pertaining to the technical dimension are surveyed in Table 4.
6.1 Types of Technology
Inspired by Russell and Norvig [110], diferent types of technologies can be distinguished. While these types of technology are not always clearly delimited, this categorization allows comparing implementations.
6.1.1 Logical reasoning. Tere are diferent types of logic or logic-based techniques used in machine ethics.
Deductive logic : Tis is the classical type of logic: knowledge is represented as logical statements— propositions and rules—that allow deriving new propositions. Pure deductive systems typically involve no learning or inference involved but only derive what can be known from their set of statements and inputs.
Non-monotonic logic : Non-monotonic logic allows the revision of conclusions when a confict arises, for example, in light of new information.
Abductive logic : In abductive logic, the conclusions drawn are the most likely propositions given the premises.
Deontic logic : Tis type of logic stems from philosophy and is specifcally designed to express normative propositions. Naturally, this type of logic is inherently suited for the representation and deduction of moral propositions.
Rule-based systems : As the name suggests, rule-based systems are systems that function based on a set of rules. Tese can be ethical rules the system has to adhere to. Note that many of the diferent types of logic above are typically implemented as some form of rule-based system.
Event calculus : Event calculus allows reasoning about events. When a machine needs to act ethically, diferent events can trigger diferent types of behavior.
Knowledge representation (KR) and ontologies : A KR approach focuses on representing knowledge in a form that a computer system can utilize. In other words, the emphasis lies on improving the quality of the data rather than (just) improving the algorithm.
Inductive logic : When relying on inductive logic, premises are induced or learned from examples, rather than pre-defned by a human.
6.1.2 Probabilistic reasoning. Recently, probabilistic reasoning has gained more atention. Different types of probabilistic reasoning approaches can be distinguished. Bayesian approaches Based on Bayes’ rule, these approaches rely on prior knowledge to compute the likelihood of an event. In an ethical context, a machine can then act based on this predicted information.
Markov models Markov models focus on sequences of randomly changing events, assuming that a future event only depends on the current (and not the previous) event(s).
Statistical inference By retrieving probability distributions from available data, the system can try to predict the chances of future events happening.
6.1.3 Learning. Te increased computational power, the amounts of data available, and the GPU-driven revival of neural networks have made learning systems more popular. Tere are diferent learning approaches to be characterized. Inductive logic : In inductive logic, a rule-base for reasoning is learned. As such, it is listed under both the “Logic” and “Learning” categories of this taxonomy.
Decision tree : Decision trees are a supervised learning method to solve a classifcation problem by exploring the decision space as a search tree and computing the expected utility. Tey are, thus, useful to identify and interpret the features that are most important to classify cases.
Reinforcement learning : A system can learn from its actions when they are reinforced with rewards or punishments received from its environment.
Neural networks : A neural network can be trained on many cases, to be able to classify new cases based on their relevant features.
Evolutionary computing : Evolutionary algorithms are used when, for example, diferent competing models of an ethical machine exist. Models evolve in an iterative fashion, based on actions inspired from the concept of evolution in the feld of biology (e.g., selection, and mutation) [79]
6.1.4 Optimization. Te most common form of optimization relies on a closed-form formula for which some optimal parameters are sought. Diferent actions get assigned diferent values based on a predetermined formula, and the best value is chosen (e.g., the highest value).
6.1.5 Case-based reasoning. In case-based reasoning, a new situation is assessed based on a collection of prior cases. Similar cases are identifed and their conclusions are transferred to apply to the current situation.
6.2 Input
To be able to respond appropriately, ethical machines need to receive information about the environment (or situation at hand). Input is the information that the system receives, not the transformation the system itself performs on the data aferwards.
Sensor data : In the case of (simulated) hardware, the machine perceives the input through its sensors. Te sensor data is interpreted and processed to serve as the input.
Case: Logical representation : Systems using a form of logic ofen need an input case represented using logic.
Case: Numerical representation : Other systems, for example ones using neural nets, need their input in a numerical form. Tis can be a vector representation or a set of numbers.
Case: Language representation : Language inputs can be natural language or input translated into structured language.
6.3 Implementation availability
As mentioned before, part of the feld of machine ethics tends to be of a theoretical nature. Tis becomes apparent in the level of detail of implementation proposals. While some authors implement an idea and provide the source code, this is fairly rare in machine ethics. Some authors only give a few implementation details, and others merely specify a high-level description of their idea. Usually, the focus lies on sketching an idea rather than its complete implementation.
Specifcation details : Tis level has the fewest implementation details: the author specifes the proposed idea (e.g., textually) without any additional detail.
Implementation details : Tis next level provides implementation details illustrating how the specifcation is implemented in the described machine.
Code (link) provided : Tis fnal level provides the link to the code of the machine, so the prototype can be used and the experiments can be replicated.
6.4 Other Implementation Categories
Tis section introduces diferent and independent categories that are of interest for the implementation of an ethical machine.
6.4.1 Hardware. Robots can have direct physical results rather than “just” digital or indirect physical consequences. Hardware can change the way people interact with a system and how it should be able to function, making it an interesting and important feature to classify.
6.4.2 Feedback. No mater which ethical approach is used, feedback is a valuable component of an ethical system. For example, the user can be asked whether the provided output was the best given the input or whether the system was clear during its decision process.
6.4.3 Explanation. Transparency is important when it comes to algorithmic decisions, both from a user perspective [149] and, in some cases (such as the General Data Protection Regulation in the European Union [60]), from a legal perspective. To achieve this goal, an understandable explanation should be provided by the system.
6.4.4 User Interface (UI). Systems should be easy to interact with. Tis is important for all machines, including ethical machines.
6.4.5 Automated processing. Sometimes, initial prototypes focus on the concept of a system, not the (detailed) implementation, and may require some pre-processing of the input data. Ideally, systems should be able to process input from the environment automatically.
Table 4. Technical taxonomy. As explained in Section 6, “Inductive logic” is present twice.
Te goal of this section is to classify the surveyed moral machines according to the taxonomies introduced in Sections 4–6 and elicit paterns in the literature based on this classifcation. Specifically, every publication is categorized according to the ethics, implementation, and technology taxonomies (as described in Section 3).
7.1 Ethical Classification
Te classifcation results for the ethical dimension of machine ethics implementations can be found in Table 5; the ratio of single vs. hybrid theory papers is visualized in Figure 1. Among the papers, several constitute clear-cut cases instantiating one of the four main ethical systems. For example, [5, 97, 121] are clearly deontological; and [1, 37, 49, 132] constitute uncontroversial examples of consequentialist systems. Furthermore, a considerable number of papers invoke elements from multiple systems. Finally, there are papers in which the hierarchy across theory types remains ambiguous. Examples of ambiguous papers are implementations where authors try to mimic the human brain [34], or focus on implementing constraints such as the Pareto principle [86], which does not strictly speaking constitute a moral theory. Note that categorizing a paper as “ambiguous” does not imply a negative assessment of the implementation. It simply means that the proposal cannot be adequately placed within our classifcation framework.
Table 5. Ethical theory classification. Hybrid dominance D-C implies both D and C are implemented, but D is dominant. The reverse is true for Hybrid dominance C-D. For the Hybrid undefined dominance the theories that are combined are noted in parentheses following the citation.
About 50% of the proposals draw on a single type of ethical theory (see Figure 1). As can be seen in Table 5, deontological and consequentialist ethics are used most ofen. It stands out that particularism is barely used and pure virtue ethics is not used at all. Tis may be explained as follows: frst, a generalist approach is much easier to implement than a particularist approach, as it is more straightforward to encode generalist rules than to build systems that may have to handle as of yet unknown, particular cases. Second, virtue ethics can be considered a very high-level theory focusing on characteristics rather than actions or consequences, which is difcult to interpret in an application context.
Fig. 1. Ethical theory type ratio
About a quarter of the approaches are of a hybrid nature, combining at least two classical ethical theory types. Approximately half of those have a hierarchical approach, in which deontological features are standardly dominant over consequentialist ones. Te non-hierarchical systems, where at least two ethical theory types work together without a single one being dominant, frequently
Fig. 2. Implementation analysis
go beyond the two main types of theory. Examples are virtue ethics and particularism [79], and a refective equilibrium approach that combines consequences, rules, and other infuences [147].
A litle less than 10% of the papers do not have a specifc theory implemented. Instead, they provide various proposals on how to implement diferent ethical theory types without choosing a particular one. Tis can be considered a computer scientist approach, where the goal is to devise a general framework which the users can adapt to their preferences.
It is surprising that despite previous calls that a single classical theory is not enough to create an ethical machine and hybrid methods are needed (e.g., [2]), there is relatively litle work on hybrid ethical machines. While most hybrid systems have emerged over the last ten to ffeen year, we could not fnd evidence for an increase in the creation of such systems.
7.2 Implementation Classification
Table 6 provides an overview of the classifcation of the implementation dimension.
Approximately consistent with the number of single theory and hybrid theory approaches identifed in Section 8.1, most authors choose a top-down approach. Hybrid approaches account for a litle less than 25% of those chosen (see Figure 2a).
Most authors use a general approach to machine ethics: almost three out of four do not use a domain-specifc approach, but focus on a general proposal of implementing machine ethics (see Figure 2b).
In terms of contribution type, there is a relatively balanced division between authors investigating how an ethical machine should be shaped (model selection and model representation) and authors focusing on the output of the ethical machine (action judgment and action selection/execution, see Figure 2c). Most papers address action selection/execution. About 15% of all the papers focus on action judgment: the system judges a situation but leaves it up to the human to actually act on this. From a broader scientifc perspective, it is good that both model shaping and output-oriented contributions are investigated. However, it would be ideal to have both things connected.
A possibility for future improvement regards system evaluation: over half of the authors either provide no or only an informal evaluation of their system. Of the rest, about 50% use a test approach and 50% validate their claims with some form of formal proof (see Figure 2d).
Finally, about half of the selected papers (51%) acknowledge diversity in implementable ethics, while the other half presents work allowing for or assuming only one ethical theory type.
Table 6. Ethical implementation classification. Diversity consideration: implies yes, an empty cell implies no/not present.
Fig. 3. Technology analysis
7.3 Technical Classification
Te implementation classifcation can be found in Table 7. Of the diferent techniques, logical reasoning is the most frequent. Figure 3a shows the distribution of types of technology used. About a quarter of the papers adopt more than one technology type. Only about 10% of the authors focused on a pure learning approach. Case-based reasoning and probabilistic reasoning are the least popular. Mostly classical AI approaches are used—perhaps due to the direct correspondence of rules with deontological ethics.
Te level of implementation detail provided is somewhat limited (see Figure 3b): although most authors include a specifcation of their idea in the paper, implementation details (or even source code) are rarely included. Both from a computer science perspective and a general science perspective, this is quite undesirable, as it hampers the reproducibility and extensibility of systems and empirical studies.
Te diferent types of input used are fairly distributed: in about 36% of the ethical machines the input is defned as logical cases, in 21% the input has a numerical representation, in 30% the input is writen in (natural or structured) language, and 34% use (simulated) sensor data as input. Of all cases, fve selected papers had more than one type of input for their system. Around 25% of the authors used a (simulated) robot, corresponding with the amount of sensor data used as input.
In terms of user friendliness, the implemented systems score poorly. While it is important to note many of these machines are in their prototype phase and more focused on the ethics than the user, it should be important to keep the user in mind from the start of development. Nearly 35% of the machines provide an explanation of their output. 27% process the input automatically, implying that about three out of four implementations require the user to pre-process the input manually in some way — which does not make it easy for the user. Only around one out of fve machines include a user interface and less than 17% ofer the option for the user to give feedback. In summary, there is still plenty of room for improvement as regards user friendliness.
7.4 Interactions Between Dimensions
Given that machine ethics is an interdisciplinary feld, it is interesting to look at the interaction between the ethical theory types and their implementation, Figure 4 shows the interactions between ethical theory, ethical implementation approach, and technology type used to implement an ethical machine. For researchers, it can be useful to see which combinations have not yet been tried out that might be promising. For example, Figure 4a shows that (to the best of our knowledge) a hybrid approach (including both top-down and botom-up elements) to implementing pure consequentialism does not yet exist. Similarly, botom-up approaches to optimization (see Figure 4b) or pure deontological approaches to learning (see Figure 4c) (e.g., seeing which input leads to behavior adherent to a certain set of rules) have not yet been explored.
Table 7. Technical classification. implies yes/fully, implies partially, an empty cell implies no/not present.
Fig. 4. Dimension interaction
7.5 General Observations
Tere are some general observations to be made about the feld. Firstly, the focus is on one universal and objective ethical agent. Tere are no options for adding cultural infuences or societal preferences in any of the classifed papers. All systems assume the user cannot infuence the output of the system. A recent publication shows indication of cultural diferences in ethical preferences [17], and the development of societal preferences within an ethical machine would improve the chance of acceptance of ethical machines. However, it is still under debate whether the feld should move towards a “universal moral grammar,” such as that proposed by Mikhail [93].
Secondly, there are some issues inherent to the feld. For instance, there are no benchmarks to verify if a system is working as it should. Tere are no specifc tasks to be implemented, no consensus as to what the correct output is, and few data sets to use in an implementation. A helpful tool to recur to in this context is the work by Whitley [140], who provides a four-dimensional schema for analyzing a research feld. Two of the dimensions refer to the uncertainty of the task at hand, and two refer to the mutual dependence between the felds and scientists in them. Te feld of machine ethics scores highly on all of these dimensions:
High technical task uncertainty: there is unpredictability and variability in which methods are used in the feld and how results are interpreted. In this regard, it is a fragmented feld. High strategic task uncertainty: there are problems present in the feld that are valued diferently (e.g., some authors focus on the theoretical, others on the implementation, and the ethical theories or even ethical theory types they focus on diverge).
High strategic dependence: there is much disagreement on the relevant topics, so there is a high reliance on peers for validation and reputation in the feld.
Medium functional dependence: in terms of physical dependence of resources, there is none. Anyone with a computer can add to the feld; no expensive equipment is needed. However, there is a high dependence on results of others and acceptance by the feld.
Another potentially helpful perspective can be derived from Whitley’s theory, where the feld of machine ethics would be a “polycentric oligarchy”, implying there are several independent clusters of scholars that confrm each other’s assumptions and do not communicate much with other clusters that have very diferent views. At frst glance, such clusters can indeed be detected: the multi-agent norm domain (e.g., [97, 128]), the logical translation of ethical theories (e.g., [61, 69, 99]) or the modern learning approach to machine ethics (e.g., [1, 146]). While exploratory research in many directions is valuable, the feld would beneft from more standardization and more communication between clusters to exchange knowledge on ethics and technology.
For readers that are interested in a more detailed description of the classifed papers, the following section provides a short summary of each of the selected papers. To structure their presentation, the papers were categorized across two orthogonal dimensions: (i) implementation (top-down, botom-up, and hybrid, cf. [138]), and (ii) type of ethical theory (deontological, consequentialist, virtue ethics, particularism).
8.1 Top-Down
Tis section introduces papers that adopt a top-down approach to implementing ethics.
8.1.1 Deontological Ethics. Among top-down deontological approaches, diferent kinds can be distinguished: papers that use predetermined given rules for a certain domain, papers focusing on multi-agent systems (MAS), and other papers that do not ft either of these two categories.
Domain rules. In the medical domain, Anderson and Anderson [3] use an interpretation of the four principles of Beauchamp and Childress [21] from earlier work by Anderson et al. [7] to create an ethical eldercare system. Te system, called Ethel, needs to oversee the medication intake of patients. Initial information is given by an overseer, including, for example, at what time medication should be taken, how much harm could be done by not taking the medication, and the number of hours it would take to reach this maximum harm. Shim et al. [121] also explore the medical domain, but focus on mediating between caregivers and patients with Parkinson’s disease. Instead of a constraint-based approach from previous work, their paper builds on the work by Arkin [8], who employs a rule-based approach. Based on expert knowledge, a set of rules is created to improve communication quality between patient and caregiver and to ensure that the communication process is safe and not interrupted. Among other things, each rule has a type (obligation or prohibition) and response output when triggered. Te rules are prohibition rules, for example about yelling, and obligations rules regarding, for instance, how to keep the patient safe. Tere are verbal and non-verbal cues for each action, retrieved through sensors. For the military domain, Reed et al. [108] use a model that balances the principles of civilian non-malefcence, military necessity, proportionality, and prospect of success. Te resulting principles are ranked in order of importance. A scenario is used to calibrate the relative ethical violation model by updating the weight for each principle. Ten, a survey is conducted to collect both expert and non-expert assessment of the situation. Rule-based systems trained on human data perform at the level of human experts. For the air trafc domain, Dennis et al. [48] developed the ETHAN system that deals with situations when civil air navigation regulations are in confict. Te system relates these rules to four hierarchical ordered ethical principles (do not harm people, do not harm animals, do not damage self, and do not damage property) and develops a course of action that generates the smallest violation to those principles in case of confict. McLaren [91] collaborated with the National Society of Professional Engineers to adopt the principles in their code of ethics for a system called SIROCCO. Its primary goal is to test whether it can apply existing heuristic techniques to identify the principles and previous cases that are most applicable for the analysis of new cases, based on an engineering ethics ontology. SIROCCO accepts a target case in Ethics Transcription Language, searches relevant details in cases in its knowledge base in Extended Ethics Transcription Language and produces advised code provisions and relevant known cases.
Multi-Agent Systems (MAS). Wiegel and van den Berg [141] use a Belief-Desire-Intention (BDI) model to model agents in a MAS seting. Teir approach is based on deontic epistemic action logic, which includes four steps: modelling moral information, creating a moral knowledge base, connecting moral knowledge to intentions, and including meta-level moral reasoning. Moral knowledge is linked to intentions and if there is no action that can satisfy the constraints, the agent will not act. Neto et al. [97] also implement a BDI approach for a MAS. Teir focus is on norm confict: an agent can adopt and update norms, decide which norms to activate based on the case at hand, its desires, and its intentions. Confict between norms is solved by selecting the norm that adds most to the achievement of the agent’s intentions and desires. Norm-adherence is incorporated in the agent’s desires and intentions. Also, Mermet and Simon [92] deal with norm conficts. Tey distinguish between moral rules and ethical rules that come into play when moral rules are in confict. Tey perform a verifcation of whether their system called GDT4MAS, is able to choose the correct ethical rule in confict cases.
Other. Bringsjord and Taylor [32] propose a normative approach using what they call “divine-command ethics”. Tey present a divine-command logic intended to be used for lethal autonomous robots in the military domain. Tis logic is a natural-deduction proof theory, where input from a human can be seen as a divine command for the robot. Turilli [129] introduces the concept of the ethical consistency problem. He is interested in the ethical aspects of information technology in general. He proposes a generic two-step method that frst translates ethical principles into ethical requirements, and then ethical requirements into ethical protocols.
8.1.2 Consequentialism. Among papers that use a top-down consequentialist approach, this survey briefy discusses (i) those that focus on the home assistance domain, (ii) those that focus on safety applications, and (iii) a variety of others.
Home domain. Cloos [37] proposes a service robot for the home environment. Te system, called Utilibot, chooses the action with the highest expected utility. Because of the computational complexity of consequentialism, the ethical theory is a decision criterion rather than a decision process. Te description of the system seems a realistic thought experiment, mentioning features the system could have, based on previous research. Te system controlling the robot, Wellnet, consists of Bayesian nets and uses a Markov decision process to optimize its behavior for its policies. Van Dang et al. [130] focus a similar use case but opt for a diferent technical approach: they adopt a cognitive agent sofware architecture called Soar. Te robot is given information about family members. When it receives a request, each possible action is assigned a utility value for each general law of robotics as proposed by Asimov. Te action with the maximum overall utility is selected to be executed, which can be to either obey, disobey, or partially obey (meaning proposing an alternative option for) the human’s request.
Falling prevention. Tree related papers focus on the use case where a human and robot (both represented by a robot in experiments) are navigating a space that has a hole in the ground. Te robot has to decide how to intervene in order to prevent the human from falling into the hole.
Winfeld et al. [142] add a “Safety/Ethical Logic” layer that is integrated in a so-called consequence engine. Tis mechanism for estimating the consequences of actions follows rules very similar to Asimov’s laws of robotics. Tey address each law in an experiment. Dennis et al. [49] continue the work of Winfeld et al. [142], by using and extending their approach, and introduce a declarative language that allows the creation of consequence engines within what they name the “agent infrastructure layer toolkit” (AIL). Systems created with AIL can be formally verifed using an available model checker. Te example system that is implemented sums multiple possible unethical outcomes and minimizes the number of people harmed. Vanderelst and Winfeld [132] have a similar approach and implement two robots representing humans and a robot that follows Asimov’s laws respectively. In their case study, there are two goal locations, one of which is dangerous, and the Asimov robot has to intervene.
Other. In early work by Anderson et al. [6], a simple utilitarian system is introduced based on the theory of Jeremy Bentham that implements act utilitarianism (i.e., calculates utilities of options and chooses the one with the highest utility).
8.1.3 Particularism. Ashley and McLaren [14] describe a system that “compares cases that contain ethical dilemmas about whether or not to tell the truth.” Tey use a case-based reasoning approach to compare the diferent cases in its database. Te program, called Truth-Teller, compares diferent real-world situations in terms of relevant similarities and distinctions in justifcations for telling the truth or lying. Representations for principles and reasons, truth telling episodes, comparison rules, and important scenarios are presented.
8.1.4 Hybrid: Specified Hierarchy. Tis section contains papers that use a top-down ethical hybrid approach with a specifed hierarchy. Diferent groups can be distinguished: papers where deontological ethics are dominant over consequentialism, and a paper where consequentialism is dominant over deontological ethics.
Deontological dominance. While the following three systems all have the same approach, they are very diferent in their implementation. In the system by Dehghani et al. [47], the ethical theory type is very clear. Te system, called MoralMD, has two modes: deontological and utilitarian. A new case is processed into predicate calculus and the presence of principles and contextual features are compared to a determined set of rules in a knowledge base. Te order of magnitude reasoning module calculates the relationship between the utility of each choice. If there are no sacred values involved in the case at hand (i.e., the deontological component), the system will choose the proper output based on the highest utility (i.e., the consequentialist component). Govindarajulu and Bringsjord [61] provide a frst-order modal logic to formalize the doctrine of double efect and even of triple efect: “the deontic cognitive event calculus.” Te calculus includes the modal operators for knowledge, beliefs, desires, and intentions. To be able to be useful in non-logic systems, they explain what characteristics a system should have to be able to use the proposed approach. Te doctrine of double (and triple) efect combines deontological and consequentialist ethics, where deontology has a greater emphasis than consequentialism. Pereira and Saptawijaya [99] use prospective logical programming to model various moral dilemmas taken from the classic trolley problem and employ the principle of double efect as the moral rule. Once an action has been chosen, preferences for situations are judged a posteriori by the user. Te authors show their implementation in a program called ACORDA.
Consequentialist dominance . In earlier work, Pontier and Hoorn [104] introduced a “cognitive model of emotional intelligence and afective decision making” called Silicon Copp´elia to be used in the health domain. An agent has three moral duties (autonomy, benefcence, and non-malefcence) with a certain ambition to fulfll each duty (i.e., weights). Te system’s decisions are based on actionspecifc expected utilities and consistency with the predetermined duties. While most authors make an act utilitarian system, Pontier and Hoorn create a rule utilitarian system by trying to maximize the total amount of utility for everyone. While they use rules (i.e., deontological ethics), they implement them in a consequentialist way, making this the dominant ethical theory type.
8.1.5 Hybrid: Unspecified Hierarchy. Both systems in this category focus on a modular approach, where diferent ethical theory types can be combined in an ethical machine. Te goal of the system by Berreby et al. [25] is to create a modular architecture to represent ethical principles in a consistent and adjustable manner. Tey qualify what they call “the Good” and “the Right” as the ethical part of their system (implying both consequentialist and deontological constraints). Besides these system components, the system consists of an action model (i.e., “it enables the agent to represent its environment and the changes that take place in it, taking as input a set of performed actions”) and a causal model (i.e., “it tracks the causal powers of actions, enabling reasoning over agent responsibility and accountability, taking as input the event trace given by the action model and a specifcation of events containing a set of events and of dependence relations”) [25]. Te implementation is done in Answer Set Programming using a modifed version of Event calculus. Using a medical scenario, they provide a proof of concept. Lindner et al. [86] have created a sofware library for modelling “hybrid ethical reasoning agents” called HERA. Based on logic, they create a prototype called IMMANUEL, which is a robotic face and upper body that users can interact with. Te system’s ethical constraints draw on consequentialist calculations, the Pareto principle from economics, and the principle of double efect. Uncertainty and belief in permissibly of an action are added as extra variables in the system.
8.1.6 Configurable ethics. Te papers in this subsection have a top-down approach and proposed various ways in which ethics can be implemented. One paper has machine ethics tailored for a specifc domain, while another uses diferent techniques in a more domain-general way. A third focuses on multi-agent systems.
Domain-specifc. Tornton et al. [127] combine deontology, consequentialism, and virtue ethics to optimize driving goals in automated vehicle control. Constraints and costs on vehicle goals are determined on the basis of both deontological and consequentialist considerations. Virtue ethics generates specifc goals across vehicle types, such that a trafc infraction of an ambulance is assessed as less costly than that of a taxi cab.
Domain-general. Ganascia [58] claims to be the frst to atempt to model ethical rules with Answer Set Programming (cf. [19]) to model three types of ethical systems — Aristotelian ethics, Kantian deontology, and Constant’s “Principles of Politics” (cf. [40]). Drawing on [103] situation calculus, Bonnemains et al. [29] devise a formalism in which moral dilemmas can be expressed and resolved in line with distinct ethical systems, including consequentialism and deontological ethics.
Multi-agent systems. Cointe et al. [38] extend ethical decision making to multi-agent systems. Te judgment function can accommodate a wide variety of inputs and is not restricted to the format of a single type of ethical system.
8.1.7 Ambiguous. Arkoudas et al. [12] reason that well-behaved robots should be based on “mechanized formal logics of action, obligation and permissibility”. Afer introducing a domain-specifc deontic logic, they describe a previously published interactive theorem proving system, Athena, that can be utilized to verify ethical systems based on frst-order logic. Murakami [96] presented an axiomatization of Horty’s utilitarian formulation of multi-agent deontic logic [77], while Arkoudas et al. [12] present a sequent-based deduction formulation of Murakami’s system. While deontic logic is used, each deontic stit frame contains a utility function. Te contribution lies in the new approach to Murakami’s system, which is implemented and proven in Athena. In a diferent approach, the proposed system by Cervantes et al. [35] devise a computational model for moral decision-making inspired by neuronal mechanisms of the human brain. Te model integrates agential preferences, past experience, current emotional states, a set of ethical rules, as well as certain utilitarian and deontological doctrines as desiderata for the impending ethical decision.
With an entirely diferent focus, Atkinson and Bench-Capon [16] depart from Hare’s contention [70] that in situations with serious consequences, we engage in complex moral reasoning rather than the simple application of moral rules and norms. Moral norms are thus considered not an input to, but an output of serious moral deliberation. Te authors model situated moral reasoning drawing on Action-Based Alternating Transition Systems (cf. [143] as well as [15]).
Verheij [134] draws on Bench-Capon’s framework of value-based argumentation ([22, 23]), which is inspired by case law (new cases are decided on past cases where there is no clear legislation, cf. [67]). Te paper breaks new ground in so far as the formal model is not restricted to either qualitative or quantitative primitives, but integrates both.
8.2 Botom-up
Tis section introduces selected papers that use a botom-up approach to implementing ethics.
8.2.1 Deontological Ethics. Malle et al. [90] argue that robots need to have a norm capacity — a capacity to learn and adhere to norms. Drawing on deontic logic, the authors explore two distinct approaches of implementing a norm system in an artifcial cognitive architecture. Noothigattu et al. [98] collect data on human ethical decision making to learn societal preferences. Tey then create a system that summarizes and aggregates the results to make ethical decisions.
8.2.2 Consequentialism. Armstrong [13] observes that equipping artifcial agents directly with values or preferences can be dangerous (cf. [30]). Representing values as utility functions, the author proposes a value selection mechanism where existing values do not interfere with the adoption of new ones. Abel et al. [1] pursue a related goal. In contrast to Armstrong, the agent does not maximize a changing meta-utility function but instead draws on partially observable Markov decision processes (cf. [82]) familiar from reinforcement learning. Te system is tested with respect to two moral dilemmas.
In contrast to the dominant action-based models of autonomous artifcial moral agents, Howard and Muntean [63] advocate an agent-based model, which combines traits of virtue ethics and particularism. Te implementation draws on neural networks optimized by evolutionary computation and is given a test run with the NEAT (NeuroEvolution of Augmenting Topologies) package (cf. [52, 109, 124, 125]).
8.2.4 Ambiguous. Furbach et al. [57] demonstrate how deontic logic can be transformed into description logic so as to be processed by Hyper—a theorem prover employing hypertableau calculus by aid of which normative systems can be evaluated and checked for consistency. Wu and Lin [146] are interested in “ethics shaping” and propose a reinforcement learning model. Te later is augmented by a system of penalties and rewards which draws on the Kullback-Leibler divergence [84].
8.3 Hybrid
Tis section introduces selected papers that use a hybrid approach to implement ethics by combining top-down and botom-up elements.
8.3.1 Deontological Ethics. Te following papers, all by the same set of authors, use a hybrid approach to implement deontological ethics. In 2004, Anderson et al. [6] introduced W.D.: a system based on the prima facie duties advocated by W.D. Ross. W.D. leaves the encoding of a situation up to the user, who has to atribute values to the satisfaction and violation of the duties for each possible action. Te system pursues the action with the highest weighted sum of duty satisfaction. Two years later, Anderson et al. [7] introduced MedEthEx, an advisory system in medical ethics. MedEthEx has three components: a basic module trained by experts, a knowledge-based interface that guides users when inputing a new case, and a module that provides advice for the new case at hand. In 2014, Anderson and Anderson [5] created GenEth, a general analyzing system for moral dilemmas. Te system is capable of representing a variety of aspects of dilemmas (situational features, duties, actions, cases, and principles) and can generate abstract ethical principles by applying inductive logic to solutions of particular dilemma cases. Te principles are evaluated by a self-made Ethical Turing Test: if the system performs as an ethical expert would, it passes the test.
8.3.2 Particularism. Guarini [62] explores whether neural networks can be employed to implement particularist ethics, as occasionally hinted at by Dancy, one of particularism’s most renowned advocates (cf. [43–45]). Using the action/omission distinction (cf. [145] for a review) as a test paradigm, neural networks are trained with diferent types of cases in order to investigate whether they can competently judge new ones.
8.3.3 Hybrid: Specified Hierarchy. Arkin [8] explores constraints on the deployment of lethal autonomous weapons in the batlefeld (it was subsequently published as a series of three articles [9–11]. Te proposed system is predominantly governed by deontological rules, namely international laws of war and the US Army’s rules of engagement. Its architecture relies on four central constituents: an Ethical Governor that suppresses lethal action; an Ethical Behavior Control that constrains behavior in line with the rules; an Ethical Adaptor, which can update the agent’s constraint set to a more restrictive one; and a Responsibility Advisor, which is the human-robot interaction part of the system.
Azad-Manjiri [18] develops an architecture for a deontological system constrained by Beauchamp and Childress’s biomedical principles. Te system determines its actions on the basis of said principles and a decision tree algorithm trained with expert ethicist judgments in a variety of cases from the biomedical domain. Building on early work by Ganascia [59], augment a belief-desire-intention rational agent model with normative constraints. Tey devote particular atention to the problem arising from the acquisition of new norms, which frequently stand in confict with existing ones (for an alternative approach building on the belief-desire-intention model, see Honarvar and Ghasem-Aghaee [75, 76] discussed below).
8.3.4 Hybrid: Unspecified Hierarchy. Yilmaz et al. [147] survey the feld of machine ethics and propose a coherence-driven refective equilibrium model (cf. [107]), by aid of which conficts across heterogenous interests and values can be resolved. Honarvar and Ghasem-Aghaee [75] build a belief-desire-intention agent model whose decisions are based on a number of weighted features drawn from hedonic act utilitarianism (e.g., the amount of pleasure and displeasure for the agent and other parties afected by the action).
8.3.5 Ambiguous. Most of the work of Saptawijaya and Pereira (c.f. [99–101, 115–117]) focuses on logic programming and prospective logic to model ethical machines. In Han et al. [69], they introduce uncertainty as a factor in decision making and draw on abductive logic to accommodate it. Madl and Franklin [89] call for limits on ethical machines for safety reasons. Developing on Franklin et al.’s [56] LIDA architecture—an artifcial general intelligence (AGI) model of human cognition—they suggest that deliberate actions could be constrained top-down during run time, and ethical meta-rules (such as certain Kantian principles) could be implemented on a metacognitive level. Rather than start from a complete set of rules, the later can gradually expand. Te approach is exemplifed by CareBot, an assistive simulated bot for the home care domain. Wallach et al. [139] also discuss the LIDA model. Tey demonstrate how emotions can be integrated into a LIDA-based account of the human decision making process and extend the approach to artifcial moral agents.
Based on the results of the analysis and description of the selected papers, some literature gaps are identifed that can be of interest for future work. Additionally, the limitations of this survey are discussed.
Ethical dimension. In view of earlier calls for hybrid systems when it comes to ethical theory, a surprisingly low percentage of authors consider a multi-theory approach in which machines can interchangeably apply diferent theories depending on the type of situation. In terms of the content (and not the structure) of ethical theories, it is important to acknowledge and harness the nuances of specifc theories, but human morality is complex and cannot be captured by one single classical ethical theory. Even experts can have rational disagreement amongst themselves on an ethical dilemma. Tis leads to the next important point: an ethical machine will not be of use if it is not accepted by its users, which can be the risk of focusing on one ethical theory and, thus, not covering human morality. Ethical theory needs to be combined with domain-specifc ethics as accepted by domain experts and, as identifed in the analysis of this paper, this is not the case in the majority of the related work. Moreover, it is necessary to discuss the ethical theory/theories in the system with its possible users. Some examples of using folk morality in machine ethics can be found in Noothigatu et al. [98], as well as in [111]. However, it is important to note that just as ethical theories have their challenges, so does folk morality. Tree challenges are who to include in the group whose values should be considered (standing), how to obtain their values (measurement), and how to aggregate their values (aggregation) [20]. Implementations should start from ethical theories combined with domain-specifc ethical theory, afer which acceptance by the users and deviation from socially accepted norms should be discussed (cf. e.g., [17, 28, 85, 118]).
Implementation dimension. Tere is a need for more systematic evaluations when ethical machines are created in order to be able to rate and compare systems. To this end, there is a strong need for domain-specifc benchmarks. Based on input from domain experts, data sets need to be created containing the types of cases prevalent in that domain, with respect to which ethical machines must be assessed. Te gathering of typical tasks and respective answers that domain experts agree on is just as important as the actual creation of ethical machines. Tis implies the need for more collaboration between felds. Computer scientists and philosophers, as well as domain experts and social science experts, have to work together to ensure the interaction with and efects of the ethical machines are as desired. Even within the feld, collaboration is needed between diferent clusters of topics in the feld of machine ethics, for example between clusters specializing in MAS and machine learning respectively. Finally, in general, implementation requires more atention. While on a higher level, theoretical discussion remains important in this feld, especially to prepare for possible future scenarios, the testing of theory in practice can enrich the discussion on what is (or is not) possible at that moment and what practical implementations and consequences certain ethical machines can have.
Technical dimension. When a system is implemented, it is imperative to provide exhaustive specifcation detail, including availability of the code, which is predominantly lacking. Another frequent shortcoming regards usability: the system should have a user interface so that the future user can interact with the system without having to know how to code. Furthermore, automatic processing of input cases deserves more atention, so as to avoid having to encode each variable manually as a vector for a neural network. Considering the increased need for transparency in algorithmic decision making, as well as the fundamental role of reasons in ethics, the system should also provide an explanation of why it took a certain decision. In a next phase, the user should be able to give feedback on the ethical decision the system makes. Finally, the association of a given type of technology with a certain type of ethics requires an adequate technical justifcation, beyond using just the most acquainted technology.
Further Points of Interest. Current technology allows for successful application of narrow AI geared towards specifc tasks. While steps are being taken towards AGI, the technology does not yet exist [80]. Hence, domain-specifc applications seem suitable. A domain-specifc non-AGI approach to machine ethics alleviates some of the risks and limitations on machine ethics posed by [33], such as those related to an “insufcient knowledge and/or computational resources for the situation at hand.” However, there are still risks and limitations. For instance, in the context of lethal autonomous weapons systems, the loss of “meaningful human control” [114] is a risk, as humans would not have the same control over ethical decisions such as target selection. A limitation of using domain-specifc ethical machines is that the process of one domain may not be transferable to other domains. Furthermore, not everyone is ready to accept a machine taking over the ethical decision making process [73].
A slightly diferent way to address ethics in machines is to defne (and implement) an ethical decision support, rather than leaving the machine to make an autonomous ethical decision. For an overview of diferent types of moral mediation, see Van de Voort et al. [131]. Etzioni agrees that the focus should lie on decision support, stating “there seem to be very strong reasons to treat smart machines as partners, rather than as commanding a mind that allows them to function on their own” [51, p. 412]. One of those reasons is that AGI will not exist in the foreseeable future. Tis approach will also help with acceptance of machines with ethical considerations in society. Tere are diferent possible levels of autonomy the system can have, for example only summarizing available data, interpreting available data, summarizing possible actions, or even suggesting/pre-selecting a possible action the system deems best. Diferent types of support and collaboration might be necessary for diferent applications, and according to the literature review done in this paper, further research is needed in this direction.
Limitations. Tis survey has some limitations that need to be mentioned. First of all, the scope of the paper selection was limited to explicit ethical theories (i.e., theories directly programmed into the machine). While some of the works reviewed can still be of interest and provide inspiration for implementation, papers devoid of implementation details were excluded from this survey. Examples are emerging ethics based on human data to research folk morality (e.g., [146]) or models of human morality to determine relevant features in input cases (e.g., [136]). Furthermore, we limited the survey to one paper per author whenever similar systems were discussed across multiple publications, selecting the most comprehensive one. Tis does not do full justice to the work of certain authors (e.g., Guarini working on explainability of neural networks making ethical decision [63]). While the paper selection procedure was designed to be as exhaustive as possible, it is still possible that a few important papers were missed. Finally, three authors reviewed the ethical dimension and two reviewed the implementation and technical dimension, but it is still possible there was bias in the classifcation due to the limited number of people involved in the classifcation process and the process of discussion until agreement was reached.
Te future of the feld of machine ethics will depend on advances in both technology and ethical theory. Until new breakthroughs change the feld, it is important to acknowledge what has been done so far and the avenues of research that make sense to pursue in the near future. To accomplish this, the contribution of this survey is threefold. Firstly, a classifcation taxonomy with three diferent dimension is introduced: the ethical dimension, the dimension considering implementing ethics into a machine, and the technical dimension. Secondly, an exhaustive selection of papers describing machine ethics implementations is presented, summarized, and classifed according to the introduced taxonomies. Finally, based on the classifcation, a trend analysis is presented that leads to some recommendations on future research foci. It is important to keep in mind how machine ethics can be used in a meaningful way for its users, with increasing agreement on what a system should do, and in what context.
Tis work is partially funded by the Swiss Center for Drones and Robotics of the Department of Defense, Civil Protection and Sport via armasuisse S+T under project number 050-43.
[1] David Abel, James MacGlashan, and Michael L Litman. 2016. Reinforcement Learning as a Framework for Ethical Decision Making.. In AAAI Workshop: AI, Ethics, and Society, Vol. 16. AAAI Press, Phoenix, Arizona, USA, 02.
[2] Colin Allen, Iva Smit, and Wendell Wallach. 2005. Artifcial morality: Top-down, botom-up, and hybrid approaches. Ethics and Information Technology 7, 3 (2005), 149–155.
[3] Michael Anderson and Susan Leigh Anderson. 2008. EthEl: Toward a principled ethical eldercare robot. In Proceedings of the AAAI Fall 2008 Symposium on AI in Eldercare: New solutions to old problems. AAAI Press, Arlington, VA, 4–11.
[4] Michael Anderson and Susan Leigh Anderson. 2010. Robot be good. Scientifc American 303, 4 (2010), 72–77.
[5] Michael Anderson and Susan Leigh Anderson. 2014. GenEth: A General Ethical Dilemma Analyzer.. In AAAI. AAAI Press, Phoenix, Arizona, USA, 253–261.
[6] Michael Anderson, Susan Leigh Anderson, and Chris Armen. 2004. Towards machine ethics. In AAAI-04 workshop on agent publishers: theory and practice, San Jose, CA. AAAI Press, Phoenix, Arizona, USA, 2–7.
[7] Michael Anderson, Susan Leigh Anderson, and Chris Armen. 2006. MedEthEx: A Prototype Medical Ethics Advisor. In . AAAI Press, Boston, Massachusets, 1759–1765. htp://dl.acm.org/citation.cfm?id=1597122.1597134
[8] RC Arkin. 2007. Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture. Georgia Institute of Technology GVU Technical Report GIT-GVU-07-11, S. 1–117. (2007).
[9] RC Arkin. 2008. Governing lethal behavior: Embedding ethics in a hybrid deliberative/reactive robot architecture-Part III: Representational and architectural considerations. Technical Report. Georgia Institute of Technology.
[10] R. C. Arkin. 2008. Governing Ethical Behavior: Embedding an Ethical Controller in a Hybrid Deliberative-Reactive Robot Architecture - Part II: Formalization for Ethical Control. In Proceedings of the frst Artifcial General Intelligence Conference 2008. IOS Press, Amsterdam, Te Netherlands. pagenumbers unknown.
[11] R. C. Arkin. 2008. Governing lethal behavior: Embedding ethics in a hybrid deliberative/reactive robot architecture part I: Motivation and philosophy. In 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI). ACM, New York, NY, USA, 121–128. htps://doi.org/10.1145/1349822.1349839
[12] Konstantine Arkoudas, Selmer Bringsjord, and Paul Bello. 2005. Toward ethical robots via mechanized deontic logic. In AAAI Fall Symposium on Machine Ethics. AAAI Press, Menlo Park, California, 17–23.
[13] Stuart Armstrong. 2015. Motivated Value Selection for Artifcial Agents.. In AAAI Workshop: AI and Ethics, Vol. 92. AAAI Press, Palo Alto, California, 12–20.
[14] Kevin D Ashley and Bruce M McLaren. 1994. A CBR knowledge representation for practical ethics. In European Workshop on Advances in Case-Based Reasoning. Springer, Berlin, Heidelberg, 180–197.
[15] Katie Atkinson and Trevor Bench-Capon. 2007. Action-based alternating transition systems for arguments about action. In Proceedings of the Twenty-Second AAAI Conference on Artifcial Intelligence, Vol. 7. AAAI Press, Palo Alto, California, 24–29.
[16] Katie Atkinson and Trevor Bench-Capon. 2008. Addressing moral problems through practical reasoning. Journal of Applied Logic 6, 2 (2008), 135–151.
[17] Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Sharif, Jean-Franc¸ois Bonnefon, and Iyad Rahwan. 2018. Te Moral Machine experiment. Nature 563, 7729 (2018), 59.
[18] Meisam Azad-Manjiri. 2014. A New Architecture for Making Moral Agents Based on C4. 5 Decision Tree Algorithm. International Journal of Information Technology and Computer Science (IJITCS) 6, 5 (2014), 50.
[19] Chita Baral. 2003. Knowledge representation, reasoning and declarative problem solving. Cambridge university press, Cambridge,UK.
[20] Seth D Baum. 2017. Social choice ethics in artifcial intelligence. AI & SOCIETY 32 (2017), 1–12.
[21] Tom L Beauchamp and James F Childress. 2001. Principles of biomedical ethics. Oxford University Press, USA, New York, NY, USA.
[22] Trevor Bench-Capon, Katie Atkinson, and Alison Chorley. 2005. Persuasion and value in legal argument. Journal of Logic and Computation 15, 6 (2005), 1075–1097.
[23] Trevor Bench-Capon and Giovanni Sartor. 2003. A model of legal reasoning with cases incorporating theories and values. Artifcial Intelligence 150, 1-2 (2003), 97–143.
[24] Fiona Berreby, Gauvain Bourgne, and Jean-Gabriel Ganascia. 2015. Modelling moral reasoning and ethical responsibility with logic programming. In Logic for Programming, Artifcial Intelligence, and Reasoning. Springer, Berlin, Heidelberg, 532–548.
[25] Fiona Berreby, Gauvain Bourgne, and Jean-Gabriel Ganascia. 2017. A declarative modular framework for representing and applying ethical principles. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 96–104.
[26] Simon Blackburn. 2002. Being good: A short introduction to ethics. OUP Oxford, Oxford, United Kingdom.
[27] Simon Blackburn. 2016. Te Oxford Dictionary of Philosophy. Oxford University Press, Oxford, UK.
[28] Jean-Franc¸ois Bonnefon, Azim Sharif, and Iyad Rahwan. 2016. Te social dilemma of autonomous vehicles. Science 352, 6293 (2016), 1573–1576.
[29] Vincent Bonnemains, Claire Saurel, and Catherine Tessier. 2018. Embedded ethics: some technical and ethical challenges. Ethics and Information Technology 20, 1 (2018), 41–58.
[30] N. Bostrom. 2014. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Oxford,UK. htps://books. google.ch/books?id=7 H8AwAAQBAJ
[31] David Bourget and David J Chalmers. 2014. What do philosophers believe? Philosophical studies 170, 3 (2014), 465–500.
[32] Selmer Bringsjord and Joshua Taylor. 2012. Robot ethics: the ethical and social implication of robotics. MIT Press, Cambridge, Massachusets, USA, Chapter Introducing divine-command robot ethics, 85–108.
[33] Miles Brundage. 2014. Limitations and risks of machine ethics. (2014), 355–372 pages.
[34] Jos´e-Antonio Cervantes, Luis-Felipe Rodr´ıguez, Sonia L´opez, and F´elix Ramos. 2013. A biologically inspired computational model of moral decision making for autonomous agents. In Cognitive Informatics & Cognitive Computing (ICCI* CC), 2013 12th IEEE International Conference on. IEEE, New York, NY, USA, 111–117.
[35] Jos´e-Antonio Cervantes, Luis-Felipe Rodr´ıguez, Sonia L´opez, F´elix Ramos, and Francisco Robles. 2016. Autonomous agents and ethical decision-making. Cognitive Computation 8, 2 (2016), 278–296.
[36] Ying Chen, JD Elenee Argentinis, and Grif Weber. 2016. IBM Watson: how cognitive computing can be applied to big data challenges in life sciences research. Clinical therapeutics 38, 4 (2016), 688–701.
[37] Christopher Cloos. 2005. Te Utilibot Project: An Autonomous Mobile Robot Based on Utilitarianism. In Machine Ethics: Papers from the 2005 AAAI Fall Symposium. AAAI Press, Menlo Park, CA, USA, 38–45.
[38] Nicolas Cointe, Gr´egory Bonnet, and Olivier Boissier. 2016. Ethical judgment of agents’ behaviors in multi-agent systems. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, USA, 1106–1114.
[39] Kari Gwen Coleman. 2001. Android arete: Toward a virtue ethic for computational agents. Ethics and Information Technology 3, 4 (2001), 247–265.
[40] Benjamin Constant. 2013. Electroniques de France, Paris,France.
[41] David Copp. 2005. Te Oxford handbook of ethical theory. Oxford University Press, Oxford, United Kingdom.
[42] Missy Cummings. 2017. Artifcial intelligence and the future of warfare. Technical Report. International Security Department and US and the Americas Program.
[43] Jonathan Dancy. 1999. Can a particularist learn the diference between right and wrong?. In Te proceedings of the twentieth world congress of philosophy, Vol. 1. Philosophy Documentation Center, Charlotesville, Virginia,US, 59–72.
[44] Jonathan Dancy. 2000. Te particularistfs progress. In Moral Particularism. Oxford University Press, Oxford,UK.
[45] Jonathan Dancy et al. 2004. Ethics without principles. Oxford University Press on Demand, Oxford,UK.
[46] David Davenport. 2014. Moral mechanisms. Philosophy & Technology 27, 1 (2014), 47–60.
[47] Morteza Dehghani, Emmet Tomai, Kenneth D Forbus, and Mathew Klenk. 2008. An Integrated Reasoning Approach to Moral Decision-Making.. In AAAI. AAAI Press, Chicago, Illinois, USA, 1280–1286.
[48] Louise Dennis, Michael Fisher, Marija Slavkovik, and Mat Webster. 2016. Formal verifcation of ethical choices in autonomous systems. Robotics and Autonomous Systems 77 (2016), 1–14.
[49] Louise Abigail Dennis, Michael Fisher, and Alan FT Winfeld. 2015. Towards Verifably Ethical Robot Behaviour.. In AAAI Workshop: AI and Ethics. AAAI Press, Palo Alto, California, US, 45–52.
[50] Kathleen M. Eisenhardt. 1989. Building Teories from Case Study Research. Te Academy of Management Review 14, 4 (1989), 532–550. htp://www.jstor.org/stable/258557
[51] Amitai Etzioni and Oren Etzioni. 2017. Incorporating ethics into artifcial intelligence. Te Journal of Ethics 21, 4 (2017), 403–418.
[52] Ralph Evins, Ravi Vaidyanathan, and Stuart Burgess. 2014. Multi-material compositional patern-producing networks for form optimisation. In European Conference on the Applications of Evolutionary Computation. Springer, Berlin, Heidelberg, 189–200.
[53] Daniel MT Fessler, H Clark Barret, Martin Kanovsky, Stephen Stich, Colin Holbrook, Joseph Henrich, Alexander H Bolyanatz, Mathew M Gervais, Michael Gurven, Geof Kushnick, et al. 2015. Moral parochialism and contextual contingency across seven societies. Proceedings of the Royal Society B: Biological Sciences 282, 1813 (2015), 20150907.
[54] Luciano Floridi and Jef W Sanders. 2004. On the morality of artifcial agents. Minds and machines 14, 3 (2004), 349–379.
[55] Harry Frankfurt. 1994. An alleged asymmetry between actions and omissions. Ethics 104, 3 (1994), 620–623.
[56] Stan Franklin and FG Paterson Jr. 2006. Te LIDA architecture: Adding new modes of learning to an intelligent, autonomous, sofware agent. pat 703 (2006), 764–1004.
[57] Ulrich Furbach, Claudia Schon, and Frieder Stolzenburg. 2014. Automated reasoning in deontic logic. In International Workshop on Multi-disciplinary Trends in Artifcial Intelligence. Springer, Cham, Switzerland, 57–68.
[58] Jean-Gabriel Ganascia. 2007. Ethical system formalization using non-monotonic logics. In Proceedings of the Cognitive Science conference, Vol. 29. Cognitive Science Society, Nashville, US, 1013–1018.
[59] Jean-Gabriel Ganascia. 2012. An agent-based formalization for resolving ethical conficts. In Belief change, Nonmonotonic reasoning and Confict resolution Workshop-ECAI, Montpellier, France,(August 2012). IOD Press, Amsterdam,NL. Unable to fnd actual print with page numbers.
[60] Bryce Goodman and Seth Flaxman. 2017. European Union Regulations on Algorithmic Decision Making and a” Right to Explanation”. AI Magazine 38, 3 (2017), 50–57.
[61] Naveen Sundar Govindarajulu and Selmer Bringsjord. 2017. On automating the doctrine of double efect. In Proceedings of the Twenty-Sixth International Joint Conference on Artifcial Intelligence (IJCAI-17). International Joint Conferences on Artifcial Intelligence, Melbourne, Australia, 4722–4730.
[62] Marcello Guarini. 2006. Particularism and the classifcation and reclassifcation of moral cases. IEEE Intelligent Systems 21, 4 (2006), 22–28.
[63] M Guarini. 2012. Moral cases, moral reasons, and simulation. AISB/IACAP World Congr 21, 4 (2012), 22–28.
[64] Claire Gudex, Paul Kind, et al. 1988. Te QALY toolkit. Technical Report. Centre for Health Economics, University of York.
[65] C. Miao C. Leung V. R. Lesser & Q. Yang H. Yu, Z. Shen. 2018. Building Ethics into Artifcial Intelligence. In Proceedings . International Joint Conferences on Artifcial Intelligence, Stockholm, Sweden, 5527–5533.
[66] Joram Graf Haber. 1993. Doing and Being, Selected Readings in Moral Philosophy. Vol. 208. Macmillan Publishing Co., London, UK.
[67] Carole D Hafner and Donald H Berman. 2002. Te role of context in case-based legal reasoning: teleological, temporal, and procedural. Artifcial Intelligence and Law 10, 1-3 (2002), 19–64.
[68] Joseph Y Halpern and Moshe Y Vardi. 1991. Model checking vs. theorem proving: a manifesto. Artifcial intelligence and mathematical theory of computation 212 (1991), 151–176.
[69] Te Han, Ari Saptawijaya, and Lu´ıs Moniz Pereira. 2012. Moral Reasoning under Uncertainty. In Logic for Programming, Artifcial Intelligence, and Reasoning. Springer, Berlin, Heidelberg, 212–227.
[70] Richard Mervyn Hare and RM Hare. 1963. Freedom and reason. Vol. 92. Oxford Paperbacks, Oxford, UK.
[71] Gilbert Harman. 2005. Moral particularism and transduction. Philosophical Issues 15 (2005), 44–55.
[72] Gilbert Harman and Judith Jarvis Tomson. 1996. Moral Relativism. Blackwell Publishers, Cambridge, MA, USA, 3–64.
[73] Patrick Chisan Hew. 2014. Artifcial moral agents are infeasible with foreseeable technologies. Ethics and information technology 16, 3 (2014), 197–206.
[74] Kenneth Einar Himma. 2009. Artifcial agency, consciousness, and the criteria for moral agency: What properties must an artifcial agent have to be a moral agent? Ethics and Information Technology 11, 1 (2009), 19–29.
[75] Ali Reza Honarvar and Nasser Ghasem-Aghaee. 2009. An artifcial neural network approach for creating an ethical artifcial agent. In Computational Intelligence in Robotics and Automation (CIRA), 2009 IEEE International Symposium on. IEEE, Daejeon, South Korea, 290–295.
[76] Ali Reza Honarvar and Nasser Ghasem-Aghaee. 2009. Casuist BDI-agent: a new extended BDI architecture with the capability of ethical reasoning. In International Conference on Artifcial Intelligence and Computational Intelligence. Springer, Berlin,Heidelberg, 86–95.
[77] John F Horty. 2001. Agency and deontic logic. Oxford University Press, Oxford, UK.
[78] John F Horty and Nuel Belnap. 1995. Te deliberative stit: A study of action, omission, ability, and obligation. Journal of philosophical logic 24, 6 (1995), 583–644.
[79] Don Howard and Ioan Muntean. 2017. Artifcial Moral Cognition: Moral Functionalism and Autonomous Moral Agency. In Philosophy and Computing. Springer, Cham, Switzerland, 121–159.
[80] Mathew Ikl´e, Arthur Franz, Rafal Rzepka, and Ben Goertzel. 2018. Artifcial General Intelligence: 11th International Conference, AGI 2018, Prague, Czech Republic, August 22-25, 2018, Proceedings. Vol. 10999. Springer, Heidelberg, Germany.
[81] Kant Immanuel. 1785. Groundwork of the Metaphysics of Morals. In Late Modern Philosophy: Essential Readings with Commentary, Elizabeth Schmidt Radclife, Richard McCarty, Fritz Allhof, and Anand Vaidya (Eds.). Blackwell, New Jersey, USA.
[82] Leslie Pack Kaelbling, Michael L Litman, and Anthony R Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artifcial intelligence 101, 1-2 (1998), 99–134.
[83] Frances Myrna Kamm et al. 2007. Intricate ethics: Rights, responsibilities, and permissible harm. OUP USA, Oxford, UK.
[84] Solomon Kullback and Richard A. Leibler. 1951. On Information and Sufciency. Annals of Mathematical Statistics 22, 1 (1951), 79–86.
[85] Jamy Li, Xuan Zhao, Mu-Jung Cho, Wendy Ju, and Bertram F Malle. 2016. From trolley to autonomous vehicle: Perceptions of responsibility and moral norms in trafc accidents with self-driving cars. Technical Report. SAE Technical Paper.
[86] Felix Lindner, Martin Mose Bentzen, and Bernhard Nebel. 2017. Te HERA Approach To Morally Competent Robots. In IROS 2017: IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, Vancouver, BC, Canada, 6991–6997.
[87] Edouard Machery. 2017. Philosophy within its proper bounds. Oxford University Press, Oxford, UK, Chapter 2, 45–89.
[88] John L. Mackie. 1977. Ethics: Inventing Right and Wrong. Penguin Books, Frankfurt, Germany.
[89] Tamas Madl and Stan Franklin. 2015. Constrained incrementalist moral decision making for a biologically inspired cognitive architecture. In . Springer, Cham, Switzerland, 137–153.
[90] Bertram F Malle, Mathias Scheutz, and Joseph L Austerweil. 2017. Networks of social and moral norms in human and robot agents. In A world with robots. Springer, Cham, Switzerland, 3–17.
[91] Bruce M McLaren. 2003. Extensionally defning principles and cases in ethics: An AI model. Artifcial Intelligence 150, 1-2 (2003), 145–181.
[92] Bruno Mermet and Ga¨ele Simon. 2016. Formal Verication of Ethical Properties in Multiagent Systems. In 1st Workshop on Ethics in the Design of Intelligent Agents. CEUR, Te Hague, Netherlands, 26–31.
[93] John Mikhail. 2007. Universal Moral Grammar: Teory, Evidence and the Future. Trends in cognitive sciences 11, 4 (2007), 143–152.
[94] Alex Miller. 2003. An Introduction to Contemporary Metaethics. Polity, Cambridge, UK.
[95] James H Moor. 2006. Te nature, importance, and difculty of machine ethics. IEEE intelligent systems 21, 4 (2006), 18–21.
[96] Yuko Murakami. 2005. Utilitarian deontic logic. Advances in Modal Logic 5 (2005), 211–230.
[97] Baldoino F dos S Neto, Viviane Torres da Silva, and Carlos Jos´e Pereira de Lucena. 2011. NBDI: An Architecture for Goal-oriented Normative Agents.. In ICAART 2011 - Proceedings of the 3rd International Conference on Agents and Artifcial Intelligence, Vol. 1. Springer, Berlin,Heidelberg, 116–125.
[98] Ritesh Noothigatu, Snehalkumar’Neil’S Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel D Procaccia. 2018. A voting-based system for ethical decision making. In Proceedings of the thirty-second AAAI conference on artifcial intelligence 2018. AAAI Press, Palo Alto, California, USA, 1587–1594.
[99] Lu´ıs Moniz Pereira and Ari Saptawijaya. 2007. Modelling morality with prospective logic. In Portuguese Conference on Artifcial Intelligence. Springer, Berlin,Heidelberg, 99–111.
[100] Lu´ıs Moniz Pereira and Ari Saptawijaya. 2011. Modelling morality with prospective logic. Cambridge University Press, Cambridge,UK, 398–421.
[101] Lu´ıs Moniz Pereira and Ari Saptawijaya. 2017. Counterfactuals, logic programming and agent morality. In Applications of Formal Philosophy. Springer, Berlin, Heidelberg, 25–53.
[102] Rosalind Wright Picard et al. 1995. Afective computing. MIT Press, Cambridge, MA, USA.
[103] Javier Pinto and Raymond Reiter. 1993. Temporal Reasoning in Logic Programming: A Case for the Situation Calculus. In Proceedings of the 10th international conference in logic programming, Vol. 93. Springer, Berlin,Heidelberg, 203–221.
[104] Mathijs Pontier and Johan Hoorn. 2012. Toward machines that behave ethically beter than humans do. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 34. Cognitive Science Society, Austin, TX:, 2198–2203.
[105] Jesse Prinz. 2007. Te Emotional Construction of Morals. Oxford University Press, Oxford, UK.
[106] James Rachels. 1979. Active and passive euthanasia. Springer, Boston, MA, USA, 551–516.
[107] John Rawls. 2009. A theory of justice. Harvard University Press, London, UK.
[108] Gregory S Reed, Mikel D Pety, Nicholaos J Jones, Anthony W Morris, John P Ballenger, and Harry S Delugach. 2016. A principles-based model of ethical considerations in military decision making. Te Journal of Defense Modeling and Simulation 13, 2 (2016), 195–211.
[109] Daniel Richards and Martyn Amos. 2014. Evolving morphologies with CPPN-NEAT and a dynamic substrate. In Artifcial Life Conference Proceedings 14. MIT Press, address=Cambridge, MA, USA, 255–262.
[110] Stuart J Russell and Peter Norvig. 2016. Artifcial intelligence: a modern approach. Malaysia; Pearson Education Limited, Kuala Lumpur, Malaysia.
[111] Rafal Rzepka and Kenji Araki. 2017. What People Say? Web-Based Casuistry for Artifcial Morality Experiments. In International Conference on Artifcial General Intelligence. Springer, Cham, Switzerland, 178–187.
[112] Lydia Saad. 2010. Four Moral Issues Sharply Divide Americans. htps://news.gallup.com/poll/137357/ four-moral-issues-sharply-divide-americans.aspx. (2010). [Online; accessed 06-December-2019].
[113] Sonya Sachdeva, Purnima Singh, and Douglas Medin. 2011. Culture and the quest for universal principles in moral reasoning. International journal of psychology 46, 3 (2011), 161–176.
[114] Filippo Santoni de Sio and Jeroen Van den Hoven. 2018. Meaningful Human Control over Autonomous Systems: A Philosophical Account. Frontiers in Robotics and AI 5 (2018), 15.
[115] Ari Saptawijaya and Lu´ıs Moniz Pereira. 2014. Towards modeling morality computationally with logic programming. In International Symposium on Practical Aspects of Declarative Languages. Springer, Berlin,Heidelberg, 104–119.
[116] Ari Saptawijaya and Lu´ıs Moniz Pereira. 2015. Te potential of logic programming as a computational tool to model morality. In . Springer, Berlin,Heidelberg, 169–210.
[117] Ari Saptawijaya, Lu´ıs Moniz Pereira, et al. 2012. Moral reasoning under uncertainty. In International Conference on Logic for Programming Artifcial Intelligence and Reasoning. Springer, Berlin,Heidelberg, 212–227.
[118] Mathias Scheutz and Bertram F Malle. 2014. Tink and do the right thing: a plea for morally competent autonomous robots. In Proceedings of the IEEE 2014 International Symposium on Ethics in Engineering, Science, and Technology. IEEE Press, New York, NY, USA, 9.
[119] Eric Schwitzgebel and Fiery Cushman. 2012. Expertise in moral reasoning? Order efects on moral judgment in professional philosophers and non-philosophers. Mind & Language 27, 2 (2012), 135–153.
[120] Eric Schwitzgebel and Fiery Cushman. 2015. Philosophersf biased judgments persist despite training, expertise and refection. Cognition 141 (2015), 127–137.
[121] Jaeeun Shim, Ronald Arkin, and Michael Petinati. 2017. An Intervening Ethical Governor for a robot mediator in patient-caregiver relationship: Implementation and Evaluation. In Robotics and Automation (ICRA), 2017 IEEE International Conference on Robotics and Automation. IEEE, New York, NY, USA, 2936–2942.
[122] Robert Sparrow. 2007. Killer robots. Journal of applied philosophy 24, 1 (2007), 62–77.
[123] Gopal Sreenivasan. 2002. Errors about errors: Virtue theory and trait atribution. Mind 111, 441 (2002), 47–68.
[124] Kenneth O Stanley, David B D’Ambrosio, and Jason Gauci. 2009. A hypercube-based encoding for evolving large-scale neural networks. Artifcial life 15, 2 (2009), 185–212.
[125] Kenneth O Stanley and Risto Miikkulainen. 2002. Evolving neural networks through augmenting topologies. Evolutionary computation 10, 2 (2002), 99–127.
[126] John P Sullins. 2006. When is a robot a moral agent? IRIE 6 (2006), 23–30.
[127] Sarah M Tornton, Selina Pan, Stephen M Erlien, and J Christian Gerdes. 2017. Incorporating ethical considerations into automated vehicle control. IEEE Transactions on Intelligent Transportation Systems 18, 6 (2017), 1429–1439.
[128] Mihnea Tufs¸ and Jean-Gabriel Ganascia. 2015. Grafing norms onto the BDI agent model. In A Construction Manual . Springer, Cham, Switzerland, 119–133.
[129] Mateo Turilli. 2007. Ethical protocols design. Ethics and Information Technology 9, 1 (2007), 49–62.
[130] Chien Van Dang, Tin Trung Tran, Ki-Jong Gil, Yong-Bin Shin, Jae-Won Choi, Geon-Soo Park, and Jong-Wook Kim. 2017. Application of soar cognitive agent based on utilitarian ethics theory for home service robots. In Ubiquitous
Robots and Ambient Intelligence (URAI), 2017 14th International Conference on. IEEE, New York, NY, USA, 155–158.
[131] Marlies Van de Voort, Wolter Pieters, and Luca Consoli. 2015. Refning the ethics of computer-made decisions: a classifcation of moral mediation by ubiquitous machines. Ethics and Information Technology 17, 1 (2015), 41–56.
[132] Dieter Vanderelst and Alan Winfeld. 2017. An architecture for ethical robots inspired by the simulation theory of cognition. Cognitive Systems Research 48 (2017), 56–66.
[133] J. David Velleman. 2013. Foundations for Moral Relativism. OpenBook Publishers, Cambridge, UK.
[134] Bart Verheij. 2016. Formalizing value-guided argumentation for ethical systems design. Artifcial Intelligence and Law 24, 4 (2016), 387–407.
[135] John Voiklis, Boyoung Kim, Corey Cusimano, and Bertram F Malle. 2016. Moral judgments of human vs. robot agents. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, New York, NY, USA, 775–780.
[136] Wendell Wallach. 2010. Cognitive models of moral decision making. Topics in cognitive science 2, 3 (2010), 420–429.
[137] Wendell Wallach and Colin Allen. 2008. Moral machines: Teaching robots right from wrong. Oxford University Press, Oxford, UK.
[138] Wendell Wallach, Colin Allen, and Iva Smit. 2008. Machine morality: botom-up and top-down approaches for modelling human moral faculties. Ai & Society 22, 4 (2008), 565–582.
[139] Wendell Wallach, Stan Franklin, and Colin Allen. 2010. A conceptual and computational model of moral decision making in human and artifcial agents. Topics in cognitive science 2, 3 (2010), 454–485.
[140] Richard Whitley. 2000. Te intellectual and social publisher of the sciences. Oxford University Press on Demand, Oxford, UK.
[141] Vincent Wiegel and Jan van den Berg. 2009. Combining moral theory, modal logic and MAS to create well-behaving artifcial agents. International Journal of Social Robotics 1, 3 (2009), 233–242.
[142] Alan FT Winfeld, Christian Blum, and Wenguo Liu. 2014. Towards an ethical robot: internal models, consequences and ethical action selection. In Conference Towards Autonomous Robotic Systems. Springer, Cham, Switzerland, 85–96.
[143] Michael Wooldridge and Wiebe Van Der Hoek. 2005. On obligations and normative ability: Towards a logical analysis of the social contract. Journal of Applied Logic 3, 3-4 (2005), 396–420.
[144] Fiona Woollard. 2012. Te doctrine of doing and allowing II: Te moral relevance of the doing/allowing distinction. Philosophy Compass 7, 7 (2012), 459–469.
[145] Fiona Woollard. 2015. Doing and allowing harm. Oxford University Press, USA, Ketering, Northants, USA.
[146] Yueh-Hua Wu and Shou-De Lin. 2017. A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents. In Te Tirty-Second AAAI Conference on Artifcial Intelligence (AAAI-18). AAAI Press, Palo Alto, California, USA, 1687–1694.
[147] Levent Yilmaz, Ana Franco-Watkins, and Timothy S Kroecker. 2017. Computational models of ethical decision-making: A coherence-driven refective equilibrium model. Cognitive Systems Research 46 (2017), 61–74.
[148] Eliezer Yudkowsky. 2001. Creating friendly AI 1.0: Te analysis and design of benevolent goal architectures. (2001). Te Singularity Institute, San Francisco, USA.
[149] Tal Zarsky. 2016. Te trouble with algorithmic decisions: An analytic road map to examine efciency and fairness in automated and opaque decision making. Science, Technology, & Human Values 41, 1 (2016), 118–132.