Reading 28: Ethical Software Engineering
|Safe from bugs||Easy to understand||Ready for change|
|Correct today and correct in the unknown future.||Communicating clearly with future programmers, including future you.||Designed to accommodate change without rewriting.|
- Explain the ethical principles to consider when you design, build, and maintain software.
- Apply four different moral lenses as you examine the ethical ramifications of a particular system.
The three technical objectives for software in 6.031 have appeared at the top of every reading and in the discussions during every class this semester. There are many other properties of good software that we have not discussed: performance, security, usability, …. Each may be as critical to the success of a software project as correctness. The topic of this reading is another such property: ethicality.
But this reading does not prescribe a moral code, or tell you which properties are more important when you are faced with tradeoffs. Like the rest of the course, we try to give you the basic tools to prevent or fix ethical problems, just like you might prevent or fix correctness problems.
- You work with other people
- on software designed by other people
- that is used by other people
- whose use of your software affects other people.
Writing software that is correct, clear, and changeable is your professional obligation to (1) the others you work with and (2) the organization you work in. Correctness is also an obligation to (3) the users of your software, who expect it to function as described. Expanding outward from those users, we encounter (4) the other people who are affected by your software, even if they don’t use it themselves, and the wider world in which all those people live.
Starting small, one ethical issue you’re familiar with is plagiarism — we asked about it on Problem Set 0, and on every problem set reflection. The Academic Integrity at MIT handbook says: “honesty is the foundation of good academic work.” The 6.031 collaboration policy and the handbook section on Writing Code describe how plagiarism violates that obligation to honesty. Any time we use code written by others, we must attribute it to the authors and give them credit for their work.
The 6.031 collaboration policy also describes when and how sharing your work from the class is and is not allowed. Software engineering has a strong history and culture of sharing. The free software movement considers unethical any software that users are not free to run however they want, modify to suit their needs, and redistribute in both original and modified form. The open source movement advocates for sharing source code as a practical advantage over keeping it proprietary. Many pieces of software that provide the foundation for modern computing are free software (meaning freedom-preserving, not simply $0). The culture of sharing is also visible in places like Stack Overflow, where programmers have collaboratively created an enormous wealth of explanations and examples from which we all benefit.
As our next step for understanding ethical software engineering, we will use the Association for Computing Machinery (ACM) Code of Ethics and Professional Conduct. The ACM is the leading international professional organization for computer scientists. (Among other things, the ACM gives out the Turing Award, the CS equivalent of a Nobel Prize.)
This section describes obligations to be honest (1.3) and to respect authorship and copyright (1.5), similar to the discussion above. It also describes an obligation around confidentiality (1.7), especially to an employer. And it introduces a few more dimensions to consider:
Section 2, Professional Responsibilities, provides a list of obligations you could consider not only as an individual, but also when you join an organization. For example, you should “strive to achieve high quality in both the processes and products” of your work (2.1) and “maintain high standards of professional competence, conduct, and ethical practice” (2.2). But these are not goals you can achieve alone.
When a large organization builds a system that is detrimental to human well-being — it causes harm, it discriminates, it increases inequality, it violates privacy — it is tempting to look for some single bad actor, an evil mastermind behind a nefarious plot to hurt people. If that’s the case, that bad actor probably is an actor, because you’re probably watching a movie where the storytelling is driven by individual villains and heroes.
Much more likely is that the large organization builds a harmful system because it doesn’t have a structure in place to ensure that the collective effort of individual participants has an ethical result. Worse, because we live in a world and society where so many existing structures are ethically unsound — they discriminate, they perpetuate inequality, they deny autonomy, etc. — an organization that pays no particular attention to acting ethically will end up reproducing in its work the ethical failures of the society around it. (See, e.g., Prof. Susan Silbey on How Not to Teach Ethics.)
When you interview at a company, you might ask: “how do you do code reviewing?” With that question, you’re probing one part of how that company creates a high quality software process (2.1), shares technical knowledge (2.2), and provides professional feedback (2.4). You might also ask, for example:
- “how do you review new features for potential abuse?”
- or “what different people or groups do you consider, and talk to, when you design a new product?”
- or “was there a time you changed a planned feature for ethical reasons — how did you decide to do that?”
It’s important to ask how the company understands and reviews the impact of its projects on people, because merely hiring good employees and setting out on a good path is not enough. If the company hasn’t designed its own structure so that ethical questions are asked and answered and the objectives of whole teams and projects are updated in response, you should be as skeptical as if your interviewer says: “we don’t need code review, we trust every engineer to write great code on their own.”
Facebook’s API for 3rd-party developers allowed an app to harvest and misuse the personal information of all the friends of users of the app. No single engineer designed or built or deployed that API. If an engineer raised ethical questions about abuse of the data, was Facebook as an organization designed to amplify and examine those questions, or silence them?
In response to anti-vaccination misinformation, Pinterest decided that they could not rely on an algorithmic approach to promote truthful posts and demote lies. Instead, they stopped returning search results for hundreds of health-related keywords, showing info from hand-picked public health sources instead. How many employees contributed to deciding on and then implementing this policy?
Finally, Section 3, Professional Leadership Responsibilities, describes ways that leaders should build ethical processes into their organization.
In explaining the general principle to avoid harm (1.2), the Code references an item from section 3: “recognize and take special care of systems that become integrated into the infrastructure of society” (3.7). Wouldn’t we all love to build a system so wonderful and widely used that it becomes a part of everyone’s everyday lives? Perhaps you can think of some such systems (social networks, again, are the ready examples). The builders of many of today’s widely-used software systems absolutely did not recognize the need for special care. As the rate of adoption of their products went up, so too did their ethical responsibilities. They failed to anticipate and design for the societal implications of their systems. The next section will offer a procedure that you and your world-changing startup can use to do better.
It’s entertaining to ponder how self-driving cars should navigate impossible ethical tradeoffs in trolley problem scenarios. But as of this writing, the only pedestrian killed by a self-driving car died even though the software eventually saw that emergency braking was required. The system was not programmed to do the emergency braking itself: “The system design precluded activation of emergency braking for collision mitigation, relying instead on the operator’s intervention to avoid a collision or mitigate an impact.” (NTSB report, page v). Did Uber consider the ethics of that design tradeoff? Have they considered how widespread self-driving cars might affect society, and what their own responsibilties are?
Suppose police use facial recognition just to track passengers through international airports. How do the ethical questions change if they track everyone within 1 mile of the airport? Or 10 miles, or 100?
Suppose police wear body cameras to increase accountability. How do the ethical questions change if the cameras are on 1%, 10%, or 100% of the time? Or as the recordings are stored longer, or made available to more people? Or fed to that facial recognition system?
“Professional competence starts with technical knowledge and with awareness of the social context in which their work may be deployed. Professional competence also requires skill in communication, in reflective analysis, and in recognizing and navigating ethical challenges.” (ACM Code 2.2)
MIT certificate required. You can’t just use your Kerberos password to access this, you must have a certificate in your browser.
- the project’s costs and benefits
- the way they are achieved
- patterns of good and bad outcomes and processes
- considering the project as a person
You’re building a touchscreen restaurant menu tablet. User testing shows that people don’t understand what the tablet is if the screen turns off, so you keep the screen on all the time… showing flashy video ads. Which moral lens definitely reveals a problem?
Correct, clear, changeable software is a starting point. Without correctness, virtually no other desirable property is achievable, because we can’t rely on the code to do what we intended. And without clear, changeable code, multiple authors will be unable to collaborate, and even a single author will eventually lose the battle with complexity and be unable to build further.
Ethicality is no different. Wherever you go on to build software, consider and test its ethical properties just as you would any others. Do not rely on yourself or anyone else to simply “get it right.” We rely on testing to tell us whether our implementation successfully satisfies a spec, and when we put on our testing hat, we work as hard as we can to break our own code. With your ethical testing hat, consider as pessimistically as possible all the ways that what you build might be misused and abused, using the moral lenses from this reading. Fixing these bugs might be tricky, because they involve so much more than just the system and its source code. Luckily, you have plenty of experience fixing tricky bugs.