Reading 28: Ethical Software Engineering
Software in 6.031
Objectives
After this reading, you should be able to:
- Explain the ethical principles to consider when you design, build, and maintain software.
- Apply four different moral lenses as you examine the ethical ramifications of a particular system.
SFB, ETU, and RFC
The three technical objectives for software in 6.031 have appeared at the top of every reading and in the discussions during every class this semester. There are many other properties of good software that we have not discussed: performance, security, usability, …. Each may be as critical to the success of a software project as correctness. The topic of this reading is another such property: ethicality.
We presume you value SFB, ETU, and RFC. Software engineers also value those properties of software, in general.
Similarly, we presume you value the ethical properties discussed in this reading, such as fairness, avoiding harm, and protecting privacy.
But this reading does not prescribe a moral code, or tell you which properties are more important when you are faced with tradeoffs. Like the rest of the course, we try to give you the basic tools to prevent or fix ethical problems, just like you might prevent or fix correctness problems.
Ethicality
Your practice of software construction, whether as a professional software engineer or in one of the innumerable other positions where you might write software, does not take place in a vacuum:
- You work with other people
- on software designed by other people
- that is used by other people
- whose use of your software affects other people.
Writing software that is correct, clear, and changeable is your professional obligation to (1) the others you work with and (2) the organization you work in. Correctness is also an obligation to (3) the users of your software, who expect it to function as described. Expanding outward from those users, we encounter (4) the other people who are affected by your software, even if they don’t use it themselves, and the wider world in which all those people live.
We’ll start, however, with some ethical principles around writing code, before we go on to consider larger questions.
Plagiarism, copyright, and licenses
Starting small, one ethical issue you’re familiar with is plagiarism — we asked about it on Problem Set 0, and on every problem set reflection. The Academic Integrity at MIT handbook says: “honesty is the foundation of good academic work.” The 6.031 collaboration policy and the handbook section on Writing Code describe how plagiarism violates that obligation to honesty. Any time we use code written by others, we must attribute it to the authors and give them credit for their work.
The 6.031 collaboration policy also describes when and how sharing your work from the class is and is not allowed. Software engineering has a strong history and culture of sharing. The free software movement considers unethical any software that users are not free to run however they want, modify to suit their needs, and redistribute in both original and modified form. The open source movement advocates for sharing source code as a practical advantage over keeping it proprietary. Many pieces of software that provide the foundation for modern computing are free software (meaning freedom-preserving, not simply $0). The culture of sharing is also visible in places like Stack Overflow, where programmers have collaboratively created an enormous wealth of explanations and examples from which we all benefit.
A code of ethics
As our next step for understanding ethical software engineering, we will use the Association for Computing Machinery (ACM) Code of Ethics and Professional Conduct. The ACM is the leading international professional organization for computer scientists. (Among other things, the ACM gives out the Turing Award, the CS equivalent of a Nobel Prize.)
General ethical principles
Read:
ACM Code Section 1, General Ethical Principles
Ethical structures, not just ethical individiuals
Section 2, Professional Responsibilities, provides a list of obligations you could consider not only as an individual, but also when you join an organization. For example, you should “strive to achieve high quality in both the processes and products” of your work (2.1) and “maintain high standards of professional competence, conduct, and ethical practice” (2.2). But these are not goals you can achieve alone.
When a large organization builds a system that is detrimental to human well-being — it causes harm, it discriminates, it increases inequality, it violates privacy — it is tempting to look for some single bad actor, an evil mastermind behind a nefarious plot to hurt people. If that’s the case, that bad actor probably is an actor, because you’re probably watching a movie where the storytelling is driven by individual villains and heroes.
Much more likely is that the large organization builds a harmful system because it doesn’t have a structure in place to ensure that the collective effort of individual participants has an ethical result. Worse, because we live in a world and society where so many existing structures are ethically unsound — they discriminate, they perpetuate inequality, they deny autonomy, etc. — an organization that pays no particular attention to acting ethically will end up reproducing in its work the ethical failures of the society around it. (See, e.g., Prof. Susan Silbey on How Not to Teach Ethics.)
When you interview at a company, you might ask: “how do you do code reviewing?” With that question, you’re probing one part of how that company creates a high quality software process (2.1), shares technical knowledge (2.2), and provides professional feedback (2.4). You might also ask, for example:
- “how do you review new features for potential abuse?”
- or “what different people or groups do you consider, and talk to, when you design a new product?”
- or “was there a time you changed a planned feature for ethical reasons — how did you decide to do that?”
It’s important to ask how the company understands and reviews the impact of its projects on people, because merely hiring good employees and setting out on a good path is not enough. If the company hasn’t designed its own structure so that ethical questions are asked and answered and the objectives of whole teams and projects are updated in response, you should be as skeptical as if your interviewer says: “we don’t need code review, we trust every engineer to write great code on their own.”
Facebook’s API for 3rd-party developers allowed an app to harvest and misuse the personal information of all the friends of users of the app. No single engineer designed or built or deployed that API. If an engineer raised ethical questions about abuse of the data, was Facebook as an organization designed to amplify and examine those questions, or silence them?
In response to anti-vaccination misinformation, Pinterest decided that they could not rely on an algorithmic approach to promote truthful posts and demote lies. Instead, they stopped returning search results for hundreds of health-related keywords, showing info from hand-picked public health sources instead. How many employees contributed to deciding on and then implementing this policy?
With great power…
Finally, Section 3, Professional Leadership Responsibilities, describes ways that leaders should build ethical processes into their organization.
In explaining the general principle to avoid harm (1.2), the Code references an item from section 3: “recognize and take special care of systems that become integrated into the infrastructure of society” (3.7). Wouldn’t we all love to build a system so wonderful and widely used that it becomes a part of everyone’s everyday lives? Perhaps you can think of some such systems (social networks, again, are the ready examples). The builders of many of today’s widely-used software systems absolutely did not recognize the need for special care. As the rate of adoption of their products went up, so too did their ethical responsibilities. They failed to anticipate and design for the societal implications of their systems. The next section will offer a procedure that you and your world-changing startup can use to do better.
It’s entertaining to ponder how self-driving cars should navigate impossible ethical tradeoffs in trolley problem scenarios. But as of this writing, the only pedestrian killed by a self-driving car died even though the software eventually saw that emergency braking was required. The system was not programmed to do the emergency braking itself: “The system design precluded activation of emergency braking for collision mitigation, relying instead on the operator’s intervention to avoid a collision or mitigate an impact.” (NTSB report, page v). Did Uber consider the ethics of that design tradeoff? Have they considered how widespread self-driving cars might affect society, and what their own responsibilties are?
Suppose police use facial recognition just to track passengers through international airports. How do the ethical questions change if they track everyone within 1 mile of the airport? Or 10 miles, or 100?
Suppose police wear body cameras to increase accountability. How do the ethical questions change if the cameras are on 1%, 10%, or 100% of the time? Or as the recordings are stored longer, or made available to more people? Or fed to that facial recognition system?
Moral lenses
“Professional competence starts with technical knowledge and with awareness of the social context in which their work may be deployed. Professional competence also requires skill in communication, in reflective analysis, and in recognizing and navigating ethical challenges.” (ACM Code 2.2)
How can we begin to do that recognizing and navigating?
Read:
Moral Lenses
MIT certificate required. You can’t just use your Kerberos password to access this, you must have a certificate in your browser.
This reading describes a process for ethical evaluation. It applies four moral lenses, different viewpoints on a project’s positive and negative impacts:
- Outcomes
- the project’s costs and benefits
- Process
- the way they are achieved
- Structure
- patterns of good and bad outcomes and processes
- Character
- considering the project as a person
The practice exercises at the end are for you to consider on your own.
reading exercises
Compared to an interface with physical buttons, a touchscreen might be much less usable by someone who cannot see the screen.
You’re designing a touchscreen cell phone. Which moral lens leads you to develop features like iOS VoiceOver or Android TalkBack?
(missing explanation)
You’re implementing a touchscreen car center console. Which moral lens captures concerns about drivers diverting their attention to look at the screen?
(missing explanation)
You’re building a touchscreen restaurant menu tablet. User testing shows that people don’t understand what the tablet is if the screen turns off, so you keep the screen on all the time… showing flashy video ads. Which moral lens definitely reveals a problem?
(missing explanation)
Software beyond 6.031
Correct, clear, changeable software is a starting point. Without correctness, virtually no other desirable property is achievable, because we can’t rely on the code to do what we intended. And without clear, changeable code, multiple authors will be unable to collaborate, and even a single author will eventually lose the battle with complexity and be unable to build further.
Although we don’t address them in 6.031, properties like performance, security, and usability cannot be bolted on later: they must be baked into the system as part of its iterative design process.
Ethicality is no different. Wherever you go on to build software, consider and test its ethical properties just as you would any others. Do not rely on yourself or anyone else to simply “get it right.” We rely on testing to tell us whether our implementation successfully satisfies a spec, and when we put on our testing hat, we work as hard as we can to break our own code. With your ethical testing hat, consider as pessimistically as possible all the ways that what you build might be misused and abused, using the moral lenses from this reading. Fixing these bugs might be tricky, because they involve so much more than just the system and its source code. Luckily, you have plenty of experience fixing tricky bugs.
For more about the ethics and effects of computing technology on society, here is a partial list of MIT courses and resources:
- 6.033 Computer System Engineering
- 6.052 Computer Systems and Society
- 6.404 AI, Decision-making, and Society
- 6.805 Foundations of Information Policy
- 6.904 Ethics for Engineers
- Social and Ethical Responsibilities of Computing in the College of Computing