Risks of discriminating against certain groups when using big data

19. Mai 2022

This section considers the problem of bias, discrimination and unfairness in machine learning. The main point of the section is that defining what counts as bias is not only a technical matter but a normative choice. We review some approaches that are currently being developed to assess the choice of a definition of bias (or discrimination, or unfairness) from the normative point of view.

The debate on the social, political, and ethical implications of artificial intelligence, in particular machine learning, engages with normative concepts such as the fairness of predictive methods. However, the debate on what makes a prediction or a score associated with a prediction fair is far older. It can be traced back at least to the era of the Civil Rights Movement in the USA when the fairness of standardised university admission test scores was debated. That debate ceased without a clear reason before an answer was agreed upon (Hutchinson and Mitchell 2019). It was perhaps predictable that this debate could be reignited by the popularisation of statistical methods to make all kinds of decisions about people that have accompanied the development of machine learning and big data. So far, scientists have not agreed on what criterion or set of criteria an algorithm must pass to be characterised as free of bias or fair. This is an active area of research, in which linking the dots between the approaches of different disciplines is difficult but highly important. The dots that must be linked are several: between purported statistical tests of indirect discrimination based on theories of discrimination developed in law versus those developed in philosophy; between definitions of bias in data science and comparable definitions in the philosophy of science; and between definitions of fairness in machine learning and those in moral and political philosophy.

A value-free definition of “unbiased” or “fair” can be given, in the sense that the mere statement of a mathematical condition does not imply any value judgement (except possibly the judgment that it is worth spending the time to state it). Values, however, enter the debate, given that different definitions of fairness have been proposed in the field of statistics. Deciding which is appropriate, whether universally or for a specific use, is not an easy matter (Chouldechova 2017; Kleinberg et al. 2017; Barocas, Hardt, and Narayanan, incomplete work in progress). More than a single statistical condition appears to bare intuitions as stating an ideal that a fair, non-discriminatory, or bias-free prediction, test, or model, should achieve. Some of these definitions can be realised jointly but only in highly ideal circumstances—for example, by a model that predicts the future with perfect certainty—or when the base rates of the target to be predicted are distributed in identical proportions in different subpopulations (e.g., subpopulations of people who are similar in sex, gender, race, or ethnicity). The value-ladenness of the entire debate about bias is recognised by all (competent) parties in the debate as uncontroversial.[i]

The idea that bias and statistical indicators are value laden is not an arcane new concept proposed to replace more established ideas that are argued to be defective but rather rests on classical and widely used metrics. These are referred to as group-fairness criteria. Such criteria are used to measure treatment inequalities associated with individuals being members of particular groups. Although we have witnessed an explosion of new definitions in the last five years, the value-laden nature of the choice of a fairness standard or desideratum is clear even if only the simplest group metrics are considered. Even within this limited set of proposals, there are measures that are not only different but mathematically incompatible. Because of the incompatibility, statisticians and machine learning theorists are in the uncomfortable position of not being able to specify the conditions that make a statistical test fair from a purely scientific point of view. Before they do so, they have to justify morally why they choose a given definition of (un)fairness.

To be more concrete, consider the standard of equal calibration, or “test fairness” as it has been called (Chouldechova 2017). This requires that the result of a statistical test (e.g., an evaluation score) has the same meaning for members of different groups for which fairness is evaluated. The simplest example of this is when a score is intended as an indication of whether the individual has or will have some feature that is currently unknown—for example, whether an individual will graduate or avoid recidivism (criminal activity) during parole. For simplicity, we shall only consider binary predictions, where the goal is to ascertain whether an individual has (or will have) a given unknown property. Typically, the answer to this question produced by a statistical method is a score that has a value between 0 and 1. It is assumed that the score has some kind of informative relationship with the truth of the statement that the individual has the property or not. For example, the score may express the probability that the individual has a given property (belongs to the positive class). In that case, one would expect that among all people with the same score (e.g., 0.2), the score is approximately equal to the share of people who have (or will have) a certain property (e.g., 20%). It would appear unfair if, for example, among people with a score of 0.2, 10% of people from group A have the property, whereas 25% of people from group B have it. It therefore seems intuitive that a score that implies different probabilities for different people, depending on the group to which they belong, has some kind of bias and, when used, may lead to some kind of unfairness. And yet, the political discussion around the software COMPAS ended up branding a widely used software application as discriminatory (Angwin and Larson 2016a), even though the underlying statistical model satisfied precisely this criterion (Brennan, Dieterich, and Ehret 2009), and the requirement COMPAS was accused of violating was mathematically incompatible with it (Kleinberg et al. 2017; Chouldechova 2017).[ii]

COMPAS is a tool that uses a statistical model to produce scores representing the chances an inmate will re-offend when released from prison. The scores in themselves do not indicate that a certain decision should be made. For instance, assigning a probability of recidivism of 0.8 to an inmate does not imply that the inmate should be given or denied parole. The company producing the tool, however, provides normative guidance suggesting risk thresholds for grouping candidates into low-risk, medium-risk, and high-risk categories. A study conducted by ProPublica showed that the average score among people who do not recidivate differed by race. Among people who did not recidivate, black defendants were more likely to be placed in the medium- or high-risk categories than white defendants, suggesting that recidivism in this group was overestimated relative to whites. Among defendants who did recidivate, white defendants were more likely to be placed in the low-risk category than black defendants, suggesting that recidivism is underestimated among white defendants relative to black ones. Both biases favoured white defendants relative to black defendants, so the ProPublica journalists and data scientists alleged that the statistical model led judges to make discriminatory bail decisions and possibly make discriminatory parole decisions (Angwin and Larson 2016a).

The bias in question, which amounts to a violation of the parity of false negative and false positive rates between groups, is mathematically inevitable whenever different base rates of recidivism occur in different populations if the same threshold values of risk are used to decide whether to give or deny parole or bail. Journalists popularised this result by saying that “bias is mathematically inevitable” (Angwin and Larson 2016b).

How has the theoretical discussion advanced since 2016? So far, the scientific community has not achieved a consensus on which measure is generally preferable, or which moral or scientific criteria should be used to determine whether the one or the other should be employed. At the same time, a new theoretical construct, called “counterfactual fairness,” has been proposed by the machine learning community (Kusner et al. 2017). Despite having “fairness” in its name, counterfactual fairness is most easily understood as an attempt to model mathematically the existence of indirect discrimination. Notice that the recent shift to speaking more about fairness and less about discrimination is due precisely to the fact that what is discussed in data science is not discrimination in its most paradigmatic and easily detectable form, where the discriminator is fully conscious of the group of the discriminated person or at least has some degree of belief about it that plays a demonstrable role in reaching the discriminatory decision. Such cases of direct discrimination are very rare in machine learning because they are easily avoided by depriving the algorithm of all information about the protected group. Counterfactual fairness deals with this problem by using a dedicated statistical analysis to detect unequal predictions and decisions that can be causally attributed to group membership, even when the information about group membership is not available explicitly to the algorithm.

Despite the considerable number of academic papers on this approach by data scientists, philosophers of science, and moral philosophers (Glymour and Herington 2019; Herington 2020), counterfactual fairness is probably not considered the gold standard of fairness in machine learning. Although surveys are lacking, it appears that it might be less used in practice than the traditional group fairness criteria, which seem to have remained the standard. Part of the reason is that using counterfactual fairness as a standard requires data scientists to make commitments about social reality that they do not feel prepared to make. Moreover, counterfactual fairness raises a host of philosophical problems because it unavoidably presupposes some kind of theorisation of the nature of the causal interactions between social constructs (e.g., race and gender) and other features that are ordinarily used to make predictions. This is not an easy matter, to say the least. Several scholars have pointed out that it is unclear whether the causal properties attributed to such constructs in this methodology are compatible with a significant number of theories about the socially constructed reality of gender and race (as opposed to, for example, naturalistic categories such as sex, which, however, also has paths of social causation that make it difficult to distinguish from gender) (Kasirzadeh and Smart 2021; Hu and Kohler-Hausmann 2020). In addition to these problems, it is unclear whether counterfactual fairness is designed to provide a new theory of what is morally wrong with discrimination or a more general framework to be adapted to one’s preferred moral theory or the theory that proves most convincing from the normative point of view. If the former, it appears that the account is either incomplete or, if interpreted as complete as it has been presented to date, faces some obvious objections such as direct conflicts with certain philosophical theorisations of what discrimination is. If the latter, it appears that philosophical guidance for integrating moral theories of discrimination with the counterfactual criteria of fairness is still to be provided. An alternative approach consists in reasoning morally about the implicit presuppositions, in terms of prescriptive desiderata within traditional definitions of group fairness, such as statistical parity, test fairness, and balanced positive and negative rates (Heidari et al. 2019; Hertweck, Heitz, and Loi 2021; Räz 2021; Loi, Herlitz, and Heidari 2021). Few scholars have taken this approach as it requires the tight integration of mathematical modeling and modeling of theories in moral or political philosophy. It has so far produced a limited number of publications but might provide some guidance in the future. Yet another approach consists in starting with a definition of fairness from philosophy or economics and investigating what it implies for machine learning (e.g., Gummadi and Heidari 2019; Liu et al. 2021). Interestingly, this approach and the previous one may eventually converge (Heidari et al. 2019).

[i] For a useful reconstruction of the arguments, see Scantamburlo (2021).

[ii] The rediscovery (Hutchinson and Mitchell 2019) of this long-forgotten fact was made in two publications with very high citation rates (Kleinberg et al. 2017; Chouldechova 2017) that formulated the problem in ways most accessible to data scientists.

Recommendations

Policy makers should include fairness as a requirement for the design of high-stake algorithmic systems. Discrimination is not avoided by avoiding the processing of group information. This must be collectible for the sake of checking the fairness of algorithms. Anti-discrimination policies must be flexible enough to legitimate different statistical standard for different use cases.