Abstract
The literature on algorithmic fairness has concentrated on the choice amongst fairness metrics, a choice constrained by well-known impossibility results (Chouldechova 2017; Kleinberg et al. 2017). This paper argues that a prior question deserves equal attention: how the categories, variables, and objective functions that constitute an algorithmic system’s representation of its domain are chosen, and when those choices themselves discriminate? We propose a criterion for distinguishing two kinds of algorithmic discrimination. A disparity is contingent, relative to a legitimate purpose and a class of admissible designs, when an alternative design within that class serves the purpose whilst reducing the disparity; it is constitutive, relative to the same parameters, when no such alternative is available. Because contingency is an existential claim and constitutivity a negative existential over a class of designs, the two carry asymmetric obligations: contingency is shown by exhibit, constitutivity by documented exhaustion, and neither is discharged by assertion. Every constitutivity claim therefore states in advance the exhibit that would refute it, which is what makes the distinction actionable in design review, audit, and litigation rather than merely classificatory. The criterion adapts the comparative logic of the less-discriminatory alternative from disparate-impact doctrine (Laufer et al. 2025) and shifts it from the level of models and outcomes to the level of representational choices, whose distinctive rigidity we analyse through the notion of open texture (Waismann 1945; Hart 1961): computational systems fix ex ante the precisifications that human decision-makers perform case by case. Three case studies (the ImageNet person categories, tokeniser-level disparities in large language models, and automated hiring together with clinical risk scoring) are used to test, not merely to illustrate, the criterion. We conclude by proposing ontological auditing, an upstream evaluation of representational choices articulated through an Ontological Impact Statement, and by locating it within the risk management and data governance provisions of the EU AI Act.
Similar content being viewed by others
1 Introduction
In 2016, ProPublica published an analysis of COMPAS, a recidivism prediction tool used across the United States, showing that Black defendants were roughly twice as likely as white defendants to be incorrectly classified as a high risk (Angwin et al. 2016). Northpointe, the company behind the system, replied that COMPAS satisfied predictive parity: the probability of reoffending, given a high-risk score, was comparable across groups. Both claims were correct. Under unequal base rates, calibration and error-rate balance cannot be jointly satisfied by an imperfect predictor (Chouldechova 2017; Kleinberg et al. 2017). The dominant response to these impossibility results has been pragmatic: if the criteria conflict, the task is to decide which criterion to prioritise in which context (Barocas et al. 2019; Friedler et al. 2021).
This paper argues that the metric-selection debate, however it is resolved, leaves a prior question untouched. Before any fairness criterion can be applied, a set of representational choices has already been made: which categories the system recognises, how abstract constructs such as risk, merit, or health need are operationalised as measurable variables, which contextual information is retained or discarded, and which objective the system optimises. These choices are not registered by the fairness metrics that operate downstream of them, yet they can be a source of discrimination in their own right (Jacobs and Wallach 2021; Selbst et al. 2019).
This paper proposes a criterion for deciding when such representational discrimination is remediable and when it is not. We distinguish contingent from constitutive algorithmic discrimination, and we define the distinction comparatively. A disparity is contingent, relative to a legitimate purpose and a class of admissible designs, if some alternative design within that class serves the purpose whilst reducing the disparity. It is constitutive, relative to the same purpose and class, if no such alternative is available. The comparative form is borrowed from a construct that United States disparate-impact doctrine already employs, the less-discriminatory alternative (Laufer et al. 2025), and shifted one level up: from the comparison of models with respect to outcome disparities to the comparison of operationalisations with respect to the disparities they make possible.
Stating the criterion this way has three consequences that the paper develops and defends. First, the criterion is relative, not absolute: nothing is constitutive tout court, only constitutive with respect to a specified purpose, a specified measure of disparity, and a specified space of admissible alternatives (Sect. 4.3). Second, constitutivity is a negative existential claim, that no alternative exists, and is therefore defeasible: it is reached by provisional exhaustion of the alternative space and can be overturned by exhibiting a single admissible alternative (Sect. 4.4). Third, the criterion carries a burden structure inherited from the doctrine that inspires it, which specifies who must show what, and this is what makes the distinction actionable rather than merely classificatory (Sect. 4.5).
The argument proceeds as follows: Sect. 2 introduces the representational precondition of algorithmic decision-making and locates our analysis within the philosophy of scientific modelling and idealisation, conceding explicitly what is common to all abstraction. Section 3 identifies what is specific to the computational case, drawing on the notion of open texture: computational systems fix ex ante, uniformly, and at scale the precisifications that human decision-makers perform case by case and can revise in situ. Section 4, the core of the paper, states and defends the criterion. Section 5 applies it to three cases chosen to test it rather than to confirm it: one in which an apparently constitutive bias proves contingent and was in fact remediated, one that requires decomposition into a contingent and a residual layer, and one pair of cases in which the layers are entangled. Section 6 revisits the impossibility theorems in the light of the criterion, showing that they are best read as a constitutivity result relative to a narrowly bounded design class rather than as a verdict on computation as such. Section 7 draws the regulatory consequences, proposing ontological auditing as an upstream complement to the EU AI Act’s existing obligations. Section 8 states the limitations of the analysis. Section 9 concludes.
2 The representational precondition of algorithmic decision-making
2.1 Translation and representational choice
Every computational system that operates on social phenomena must perform a translation: the lived, contextual, and often ambiguous fabric of social reality must be rendered into a formal vocabulary of variables, features, and numerical values. This translation involves choices about what to measure, how to categorise, where to draw boundaries, and what to exclude. These choices are not merely methodological. They determine what the system recognises as existing and what remains invisible to it (Bowker and Star 2000; Hacking 1999).
Consider the encoding of race as a feature in a predictive model. Social scientists have long argued that race is a complex, historically contingent, relational category that resists fixed definition; a computational system nevertheless requires a discrete variable, a finite set of mutually exclusive categories with assignment rules. The gap between the social phenomenon and its computational surrogate is not a matter of imprecision to be reduced by better data collection; it reflects a choice about which distinctions are to count as relevant. Nor does the move to apparently more physical proxies dissolve the problem. In computer vision, evaluations of performance across populations frequently invoke the Fitzpatrick skin-type scale, a six-point dermatological taxonomy designed to assess phototoxic risk. The shift from race to skin type abandons the pretence that race is a measurable physical property, which is methodologically welcome, yet the new categories remain a small ordinal compression of a complex phenomenon involving appearance, lighting, camera response, and social perception. Finer granularity can reduce particular errors without changing the structural point: any computable category is the residue of a prior decision about what is to count as a relevant distinction. The measurement literature in algorithmic fairness has made this point with particular clarity: every fairness analysis presupposes an operationalisation of an abstract construct, and the validity of that operationalisation is rarely interrogated by the apparatus that consumes its output (Jacobs and Wallach 2021).
Two further observations sharpen the picture. First, computational categories do not merely register the social; they intervene in it, classifying populations in ways that feed back into the practices, institutions, and self-understandings of the classified (Hacking 2007). Models are mediating instruments, not passive mirrors (Morrison and Morgan 1999). Second, the representational choices at issue are relative to a chosen level of abstraction in Floridi’s (2008) sense: any analysis of an information system fixes, through its choice of observables, what counts as a distinction within the system. We use the method of levels of abstraction here in its intended pragmatic role, as a device for making the chosen level explicit and comparable with alternatives, not as support for any stronger claim that categories constitute reality without remainder.
We will call the gap that these choices open between a social phenomenon and its computational surrogate the ontological fracture, and we will treat the term throughout as shorthand for a family of representational costs, to be specified case by case, rather than as the name of a single metaphysical fact.
2.2 Epistemic grain and computational fog
Two epistemic consequences of the representational precondition will matter for what follows:
The first is a resolution limit. Just as the grain of a photographic film determines the finest detail it can capture, the feature space of a computational model determines the finest distinctions it can register. We call this the system’s epistemic grain. Phenomena that fall below it are not merely missed; they are unavailable in principle to any analysis conducted in the system’s own terms.
The second consequence concerns how that limit is misread from inside the system. Once categories have been fixed, whatever exceeds them registers not as evidence of an inadequate category scheme but as residual error: noisy datapoints, outliers, anomalies to be reduced by more data or better regularisation. We call the underlying failure ontological error: the inadequacy of the category system itself relative to the domain as distinct from misclassification of an individual within the system’s own terms. Quaresmini and Primiero (2024) analyse the same failure in terms of data quality as a breakdown in the completeness and consistency of the label set relative to the dataset it is meant to organise. Gender classifiers trained on a binary scheme illustrate the structure: the difficulty confronting a person whose identity does not fit the binary is not a misclassification within the scheme but an inadequacy of the scheme, which the system can only record as noise. We call the resulting second-order invisibility computational fog: the distortions introduced by the representational choices are not representable within the representation they produced, and are therefore systematically misread as measurement error still to be driven down. Section 3.2 locates this notion precisely with respect to the existing taxonomies of algorithmic opacity, to which it is related but not identical.
2.3 What is common to all modelling
It would be a mistake to present the phenomena just described as if they were unique to computation. Discretisation, decontextualisation, the loss of situational richness, and the need to fix categories in advance are general features of simplified, abstract, idealised representations of complex targets. The philosophy of science has analysed them extensively: idealisations misrepresent deliberately and productively (Cartwright 1983), models are similar to their targets only in respects and to degrees fixed by the modeller’s purposes (Weisberg 2013), and idealisation is not a regrettable departure from an ideal of completeness but a pervasive and legitimate feature of successful science (Potochnik 2017). Classification systems in bureaucracies exhibit the same costs without any computer being involved (Bowker and Star 2000), and the sociology of commensuration has documented how the reduction of qualities to common quantitative metrics transforms what it measures (Espeland and Stevens 1998).
We therefore do not claim that the representational costs of Sect. 2.1 are qualitatively unprecedented, nor that computational modelling is defective for incurring them. All modelling incurs them. The question that matters for a theory of algorithmic discrimination is narrower and more tractable: what changes when the idealised representation is fixed in executable code and deployed at scale in socially consequential decisions? The next section answers that question, and the answer is what the rest of the paper builds on.
3 What is specific to the computational case
3.1 Open texture and ex ante precisification
Waismann (1945) observed that empirical concepts are never delimited in all possible directions: however precisely a concept is defined, situations can arise in which its application is genuinely undetermined, and the concept must be sharpened on the spot. He called this open texture. Hart (1961) transported the notion into legal philosophy: general rules, however carefully drafted, exhibit an ineliminable indeterminacy at their margins, and adjudication is the practice of resolving that indeterminacy case by case, in view of the particulars, with the resolution itself remaining open to revision in the next case.
Social categories of the kind algorithmic systems encode (race, gender, merit, risk, need) are paradigmatically open-textured. Human decision-makers who apply them do what Hart’s judges do: they precisify in the presence of the particular case, drawing on context that no antecedent rule enumerated, and they retain the capacity to precisify differently when the next case differs in respects the previous precisification did not anticipate. Recent work has shown that this dynamics operates even on the technical vocabulary of algorithmic regulation itself: Baldini and De Benedetto (2025) analyse a jurisprudential case in which first-instance and appellate judges carved out contrasting notions of algorithm, and show that the contrast is best understood as two context-determined sharpenings of an open-textured concept rather than as an error by either court. Lanamäki et al. (2025) examine how organisations must reach closure on what open-textured legal requirements demand, taking the compliance architecture of the EU AI Act as their example.
Here is the specific difference we propose. A computational system does not and cannot precisify in situ. The precisification of every open-textured concept it uses is performed once, ex ante, at design time, by the choice of features, categories, thresholds, and objectives; it is then applied uniformly to every case the system encounters, at whatever scale the system is deployed, without a human interpreter positioned to notice that the present case exceeds the precisification and to re-open it. What in human institutional practice is a revisable, case-sensitive, distributed activity becomes a fixed parameter of the decision procedure. The ontological fracture of Sect. 2.1 can now be restated without metaphysical inflation: it is the closure, under a finite representational scheme fixed in advance, of concepts whose ordinary institutional use is open-textured. The costs of idealisation are common to all modelling; the ex ante, uniform, and unrevisable character of the precisification is what the computational case adds, together with the scale and automation that make the fixed precisification consequential for entire populations at once.
This restatement also explains why the familiar reassurance that a human remains in the loop underdelivers. The overseer of Article 14 of the AI Act reviews outputs produced under a precisification already fixed; what the overseer cannot do, from that position, is re-open the precisification itself, since the information that would motivate re-opening it has typically been discarded below the system’s epistemic grain.
3.2 Locating computational fog amongst the opacities
The notion of computational fog introduced in Sect. 2.2 must earn its place against an existing and well-developed literature on algorithmic opacity. Burrell (2016) distinguishes three forms: opacity as intentional corporate or state secrecy, opacity as technical illiteracy, and an opacity intrinsic to machine learning at scale, arising from the mismatch between mathematical optimisation over high-dimensional feature spaces and the demands of human-scale reasoning. Pasquale (2015) and Ananny and Crawford (2018) analyse the institutional dimension and the limits of transparency as a remedy. On the legal side, Tschider (2021) treats opacity not as a general epistemic difficulty but as a doctrinal problem: a property of AI systems that frustrates specific legal mechanisms, such as causation, disclosure, and explanation requirements.
Computational fog is not a fourth species of Burrell’s genus because it does not concern the inscrutability of the learned function at all. A system could be fully transparent in all three of Burrell’s senses (open code, expert observers, an interpretable model class) and the fog would persist, because it attaches to the input ontology rather than to the mapping: what cannot be recovered from inside the system is the set of distinctions that the choice of categories discarded before any learning occurred. In Tschider’s doctrinal register, the corresponding point is that disclosure obligations reach the system’s variables and logic but not the space of alternative operationalisations that were never built, which is precisely where Sect. 4 will locate the legally relevant comparison. Relating the notion to these taxonomies, rather than presenting it as self-standing, is what the term contributes: it names the residual opacity that survives full functional transparency.
3.3 The absent interpreter
One further feature of the computational case deserves separate statement because the case studies will repeatedly turn on it. In human decision chains, an interpreter is positioned between the representation and the decision: a clinician who reads a risk score against the patient in front of her, a recruiter who reads a CV against a career trajectory the form fields did not capture. The interpreter can re-contextualise, override, and in the limit re-describe. In deployed algorithmic systems, this position is either unoccupied or occupied by someone whose informational situation has already been shaped by the system’s grain (Sect. 3.1). The consequence is that representational choices which would be harmless in the presence of a competent interpreter, because correctable on the fly, become consequential in her absence. This is not an argument against automation; it is an argument for auditing the representational choices with a seriousness proportional to the absence of downstream correction, which is the burden Sect. 7 takes up.
4 Constitutive and contingent discrimination: a criterion
This section states the paper’s central distinction and gives it a criterion of application. We proceed in five steps: the comparative logic from which the criterion borrows (4.1); the shift of that logic to the representational level, which requires and receives an explicit justification (4.2); the parameters to which every classification is relativised (4.3); the logical form of constitutivity claims and its consequences for their epistemic status (4.4); the burden structure that makes the distinction actionable (4.5). Section 4.6 states what the criterion is not, separating it from four conflations that the surrounding literature, and the vocabulary of Sects. 2 and 3 taken alone, make easy to commit.
4.1 The comparative logic of the less-discriminatory alternative
The United States disparate-impact doctrine contains a construct whose logic we propose to generalise. When a facially neutral policy produces a disparate impact on a protected group, the defendant may justify the policy by showing that it serves a legitimate purpose; the challenger may then prevail by exhibiting a less-discriminatory alternative, an alternative policy that serves the same legitimate purpose whilst producing a smaller disparity. Laufer et al. (2025) have recently examined what this construct means when the policy is an algorithm: what it takes for one model to count as an available alternative to another, how the same business need is to be held fixed across the comparison, and how candidate alternatives can be searched for and validated.
Three features of the construct matter for our purposes, independently of its doctrinal home. First, it is comparative rather than intrinsic: nothing is discriminatory or justified in isolation, only relative to what else was available. Second, it is relativised to a purpose: the comparison holds the legitimate aim fixed and quantifies over the ways of achieving it. Third, it is existential in form: the challenger’s showing is the exhibition of one alternative, and the defendant’s is the unavailability of any. These three features, we will argue, are exactly what a workable distinction between remediable and unremediable algorithmic discrimination requires, and their absence is what leaves the distinction between remediable and unremediable discrimination, in the articulations so far available, exposed to the charge of circularity.
4.2 Shifting the comparison to the representational level
The doctrine, and the technical literature that operationalises it, compares models: given a baseline predictor, is there another predictor, trained perhaps on the same data, that meets the same business need with less disparate impact? Our claim is that the same comparative logic applies, and is needed, one level up, where the objects compared are not models but operationalisations: the choices of construct, category scheme, feature space, and objective through which the domain is rendered computable in the first place.
The shift requires justification because it is not innocent. An objection immediately arises: the less-discriminatory alternative is defined over outcome disparities, which are measured downstream, whereas representational choices are upstream of any outcome; comparing representations by their disparities seems to smuggle back in precisely the downstream perspective that the representational critique declares insufficient. The objection would be decisive against a naive transposition. Our transposition is not naive, and the justification has three parts.
First, representational choices are design choices. The developer who selects an operationalisation of health need, a category scheme for gender, or a tokenisation strategy for text is exercising the same kind of discretion, within the same design process, as the developer who selects a model class or a decision threshold. The doctrinal question, was there a less-discriminatory way of achieving the same legitimate end, does not distinguish in principle between these exercises of discretion; it quantifies over whatever the designer could have done differently. There is therefore no principled reason to confine the alternative space to post-representational choices, and Jacobs and Wallach’s (2021) demonstration that fairness properties are sensitive to the operationalisation of the constructs they are defined over shows that confining it in that way excludes some of the alternatives that matter most.Footnote 1
Second, the comparison at the representational level is a comparison of possibility spaces, not of realised outcomes. Two operationalisations are compared by the disparities they make achievable: operationalisation A dominates operationalisation B, relative to a purpose, when the best system buildable on A serves the purpose with less disparity than the best system buildable on B. The downstream measurement is thus not smuggled back in as the object of evaluation; it enters as the yardstick by which representational alternatives are ranked, which is unavoidable for any criterion that is to have empirical content at all. The alternative, ranking representations by some purely intrinsic representational virtue, is what deprives the distinction of a non-circular criterion, since fidelity to the phenomenon cannot itself be assessed except through a further representation of it.
Third, the shift is what gives the construct traction on cases the model-level version cannot reach. The empirical study of Obermeyer et al. (2019), discussed in Sect. 5.3, is the canonical demonstration: a widely deployed clinical algorithm exhibited a large racial disparity that no model-level intervention had removed, because the disparity was located in the choice of prediction target, health costs as an operationalisation of health need. Re-operationalising the target largely removed the disparity. At the model level, no less-discriminatory alternative was visible; at the representational level, one existed and was found. A criterion that cannot represent this case misses the phenomenon this paper is about.
We therefore define:
Definition (representational LDA test). Let P be a legitimate purpose, D a specified measure of disparity, and C a class of admissible operationalisations of the constructs the purpose requires, where admissibility comprises technical feasibility, lawfulness, and service of P to the degree the deployment context demands. An algorithmic disparity produced by a given operationalisation and measured by D is contingent relative to (P, D, C) if there exists an operationalisation in C on which a system can be built that serves P whilst producing a smaller disparity by the lights of D. It is constitutive relative to (P, D, C) if no operationalisation in C supports such a system.
Every term of the definition is load-bearing,Footnote 2 and the next two subsections discharge the obligations it incurs: the relativisation to (P, D, C) is examined in 4.3, and the epistemic status of the negative existential in 4.4.
4.3 Relativity to purpose and to the measure of disparity
The definition makes no sense with the parameters left free, and this is a feature rather than a concession. Hellman (2020) has argued that disagreements over which fairness metric a system should satisfy are not technical disagreements resolvable by better mathematics; they trace back to substantive disagreements about what makes discrimination wrongful, for instance whether the wrong lies in the disparate ratio of errors or in what a score is taken to say about the persons scored. If Hellman is right, and we take her to be, then D cannot be fixed by the theory; it must be fixed by the normative analysis of the deployment context, and different defensible choices of D can classify the same system differently. The same holds for P: whether an operationalisation serves the same purpose is not read off the code but argued for, and the level of generality at which the purpose is described partly determines which alternatives count as serving it, a difficulty the doctrinal literature knows well (Laufer et al. 2025). The clinical algorithm of Sect. 5.3 makes the stakes of that description concrete. If P is described as identifying the patients with the greatest health need, then operationalising need as cost serves P poorly and a demonstrated alternative is available. If P is described as identifying the patients whose future spending is most reducible by intervention, then cost is not a proxy for the target but the target itself, and the alternative does not serve P at all. Both descriptions have institutional backing in the financing of care management, and no fact about the algorithm adjudicates between them. Relativisation to P is therefore not an escape from the normative question but the point at which the criterion hands that question back, in a form that can be argued, to the parties entitled to decide it. Philosophical work questioning whether the standard statistical criteria are genuine requirements of fairness at all (Hedden 2021) reinforces the point from the other direction: if some criteria lack normative standing, then their violation cannot ground a classification of constitutive discrimination, however mathematically inevitable the violation is.
The consequence for our criterion is that classifications of the form this disparity is constitutive are always elliptical for this disparity is constitutive relative to (P, D, C), and debates that appear to be about the classification are often, on inspection, debates about the parameters. We regard the criterion’s ability to make that structure explicit as one of its principal services: it converts an apparently metaphysical disagreement, whether a bias is inherent to computation, into three tractable and separately arguable questions, what the purpose is, what the relevant disparity is, and what the admissible design space contains.
4.4 Constitutivity as a defeasible negative existential
Contingency claims and constitutivity claims are not epistemically symmetric, and the criterion makes the asymmetry explicit. A contingency claim is verified by exhibition: produce one admissible operationalisation that serves the purpose with less disparity, and the classification is settled. A constitutivity claim asserts that no such operationalisation exists in C, a negative existential over a space that is not, in general, surveyable. Except in special cases where the space is small enough to exhaust or where an impossibility result quantifies over the whole of C at once, as the fairness impossibility theorems discussed in Sect. 6 do for the narrow class they are stated over, constitutivity can therefore never be conclusively verified. It is established, when it is, by the provisional exhaustion of the alternatives that the current state of knowledge makes available, and it remains permanently open to refutation by a single new exhibit.
Three consequences follow, and we accept all three rather than resisting them. First, the honest form of most constitutivity claims is explicitly hedged: no alternative within class C is currently known, rather than none exists. Our case studies in Sect. 5 are formulated accordingly. Second, classifications are revisable with technical progress: an expansion of C, through a new technique, a new data practice, or a relaxation of a constraint previously treated as fixed, can convert a constitutive disparity into a contingent one. Section 5.1 presents a case in which a classification was revised, and we treat the reclassification as evidence that the criterion has empirical grip, not as an embarrassment. Third, the criterion states its own falsification condition: a classification of a disparity as constitutive relative to (P, D, C) is refuted by exhibiting one member of C that serves P with less disparity by the lights of D. The open-ended remark that apparent contingency may always reveal a deeper constitutive layer, which supplies no way of being wrong, is thereby replaced by a classification that specifies in advance what would show it to be wrong.
Operationalising the test inherits, and we flag rather than hide, the difficulties that Laufer et al. (2025) document for the model-level construct: candidate alternatives must be evaluated on data, evaluations on training data overstate what will generalise, and in the absence of adequate held-out data the comparison of a deployed design with a hypothetical alternative faces fundamental rather than incidental obstacles. At the representational level, these difficulties are compounded by the cost of building enough of the alternative to evaluate it. The criterion therefore does not promise cheap classifications; it promises well-formed ones, with explicit parameters and explicit defeat conditions, which is what the practices proposed in Sect. 7 require of it.
4.5 The burden structure
The doctrinal origin of the criterion supplies one further element that a purely philosophical formulation would lack: an allocation of burdens. In disparate-impact litigation, the showing moves among the parties: the challenger establishes the disparity, the defendant justifies the practice by its purpose, and the challenger answers by exhibiting a less–discriminatory alternative. Transposed to the representational level, this yields a division of labour that makes the contingent/constitutive distinction actionable in design and oversight contexts, and not only descriptively correct after the fact.
The party contesting a system, whether a litigant, an auditor, or a review committee, carries the burden of exhibition: contingency is established by producing an admissible alternative operationalisation, at a level of development sufficient to make the comparison evaluable. The party deploying the system carries the burden of exhaustion: a claim that the disparity its representational choices produce is constitutive is sustained only by documenting the alternative operationalisations considered, the grounds on which each was excluded from C or found not to serve P, and the searches undertaken. Neither burden is discharged by assertion. The Ontological Impact Statement of Sect. 7.2 is, in substance, the standing documentary form of the deployer’s burden: it is what discharging the burden of exhaustion looks like before any dispute arises.
This allocation also answers, in practical terms, the unfalsifiability worry. A deployer cannot escape scrutiny by declaring a disparity constitutive, because the declaration is only as good as the documented exhaustion behind it; and a critic cannot inflate every disparity into an indictment of computation as such, because the criticism that has purchase is the one accompanied by an exhibit. The distinction thereby stops functioning as a rhetorical endpoint, invoked to close a debate, and starts functioning as a procedure that opens one.
The elements assembled in this section can be set out as a four-step protocol, which is how an auditor or a design review would use them. First, fix the parameters: state the legitimate purpose P at the level of generality at which the deployment is defended, and the measure of disparity D that the normative analysis of the context selects, recording that both are contestable and by whom. Second, delimit the admissible class C: set out the technical, legal, and economic constraints that bound the operationalisations actually available in this setting, and justify each of them rather than assuming it, since a class drawn narrowly enough will make almost any disparity constitutive. Third, search and record: document the alternative operationalisations considered, the grounds on which each was excluded from C or found not to serve P, and the searches undertaken. Fourth, state the defeat condition: name what an opposing party would have to exhibit, namely a member of C that serves P with less disparity by the lights of D, for the classification to be withdrawn. A classification produced without the fourth step is not a classification under this criterion.
4.6 What the criterion is not
Four clarifications, each blocking a conflation that the vocabulary of Sects. 2 and 3 makes tempting, complete the statement of the criterion.
First, upstream does not mean constitutive. The location of a choice in the design pipeline settles where a disparity is introduced, not whether it is remediable. Upstream choices are often the most consequential precisely because everything downstream inherits them, but consequential and unremediable are different properties, and Sect. 5.1 exhibits an upstream choice, a category scheme inherited from a lexical taxonomy, whose discriminatory effect was straightforwardly contingent by our criterion. The slide from upstream to constitutive to irremediable is the single most productive source of misclassification in this area, and the criterion is designed to make it impossible to perform tacitly: each step requires a separate showing.
Second, the failure of one remedy does not establish constitutivity. That a particular intervention inherits or reproduces the bias it was meant to remove, as data augmentation for under-represented groups can inherit the stereotypes of the generative model that produces the synthetic data, shows that this member of C fails, not that C is empty. Arguments of this form retain illustrative value, they display the depth at which a disparity is lodged, but under the criterion they contribute to a constitutivity claim only as entries in the documented exhaustion, never as its conclusion.
Third, the criterion does not classify systems, it classifies disparities. A single system typically exhibits several disparities with different classifications, and a single disparity typically decomposes into a contingent component, removable within the current C, and a residual component for which no alternative is currently known. Section 5.2 works through such a decomposition in detail. Asking whether a system as a whole discriminates constitutively is, on our analysis, a category mistake, and one that the framing of public debate about algorithmic bias routinely invites.
Fourth, the criterion is compatible with, and indeed explains, the observation that no algorithmic system is free of representational cost. Since every operationalisation of an open-textured construct closes its texture ex ante (Sect. 3.1), every member of C carries some such cost; the criterion does not ask for a costless design, it asks whether the specific disparity at issue is reduced by some admissible design. The claim that some discrimination is constitutive is therefore not the claim that computation is irredeemable; it is the claim that, for certain disparities, certain purposes, and the currently available design space, the choice is between the disparity and the purpose, and that this fact, where it obtains, should be established, documented, and weighed openly rather than discovered after deployment.
5 Applying the criterion: three tests
The cases in this section test the criterion rather than illustrate it, and they are chosen adversarially: a case in which the criterion overturns the classification the framework’s own vocabulary would suggest (5.1), a case that resists classification until it is decomposed (5.2), and a pair of cases in which contingent and residual layers are entangled and must be separated empirically (5.3). A wider range of cases is discussed in Di Giorgio (2026).
5.1 ImageNet and WordNet: apparent constitutivity, actual contingency
The ImageNet dataset, long the standard benchmark for object recognition, was constructed on top of WordNet, a lexical taxonomy organised by hyponymy. Because WordNet contains, under the synset for person, a range of derogatory and stereotyped categories, ImageNet inherited those categories together with their visual instantiations: photographs of real people were assigned to demeaning classes because the underlying conceptual vocabulary admitted those classes (Crawford and Paglen 2021). The bias here is located exactly where our framework looks: not in mislabelled images or skewed sampling but in the category scheme through which the visual world was rendered classifiable, upstream of any training.
The vocabulary of Sects. 2 and 3, taken alone, invites the classification constitutive: the disparity is inscribed in the input ontology itself. The criterion returns the opposite verdict, and the historical record confirms it. An admissible alternative operationalisation existed: filter the person subtree, removing unsafe and offensive synsets and rebalancing the remainder, whilst preserving the dataset’s purpose as a recognition benchmark. It was not merely available in principle; it was carried out (Yang et al. 2020). Relative to the purpose of benchmarking object recognition, the disparity was contingent, and it was remediated by exactly the kind of representational intervention over which our alternative space quantifies.
We present this case first for a methodological reason. A criterion suspected of unfalsifiability is best defended by exhibiting a case in which it classifies against the theoretical temptation of the framework that proposes it. The upstream location of the WordNet bias, in the category scheme itself, makes it a natural candidate for constitutive discrimination, and the representational vocabulary of Sects. 2 and 3, taken alone, would so classify it. That classification is wrong, and demonstrably so. Upstream is not constitutive: the case establishes the first clarification of Sect. 4.6 on live material rather than by stipulation.Footnote 3
5.2 Tokenisation: decomposing a mixed case
Before a large language model processes text, the input is segmented into tokens, subword units drawn from a fixed vocabulary.Footnote 4 The dominant construction method, byte pair encoding, builds the vocabulary by iteratively merging the most frequent symbol pairs in a training corpus, so the vocabulary reflects the corpus’s statistical distribution: well-represented languages receive compact, semantically coherent tokenisations, whilst under-represented languages are fragmented into small, less-meaningful units. Petrov et al. (2023) show that the asymmetry is large, with the same content requiring up to fifteen times more tokens in some languages than in others, and that it persists even for tokenisers intentionally trained for multilingual support. Because commercial interfaces price and bound computation by the token, the asymmetry compounds into an access disparity: speakers of affected languages receive less context, shorter outputs, higher latency, and higher cost per unit of communicated meaning (Ahia et al. 2023), a compounded disadvantage concentrated in regions where access is already economically constrained.
It is tempting to classify tokeniser bias as constitutive on the ground that the greedy, frequency-based merge strategy of byte pair encoding will always produce uneven tokenisation. The formulation defeats itself. By tying the claim to one algorithm, it makes the disparity contingent by the definition of Sect. 4.2: alternative merge strategies, deliberate vocabulary allocation across languages, and byte-level tokenisation are active research directions, and a claim about what one algorithm always does cannot establish that no admissible alternative exists. What the criterion prescribes in a case of this shape is not a verdict but a decomposition.
The contingent component is everything attributable to the choice of construction method and allocation policy within a fixed vocabulary budget. Here alternatives exist and demonstrably reduce the disparity: multilingually balanced training corpora, allocation rules that guarantee per-language coverage, and related interventions. Relative to the purpose of general-purpose language modelling, this component is contingent, and responsibility for it is assignable in the ordinary way.
The residual component is what survives every currently known member of the class. The natural candidate remedy, dropping the learned vocabulary altogether in favour of character-level or byte-level encoding, is instructive because its limits are measured in the very study that documents the problem: character-level and byte-level models still exhibit more than a fourfold difference in encoding length for some language pairs (Petrov et al. 2023), reflecting the differing information densities of scripts and morphologies and, at the byte level, the differing costs at which writing systems are encoded (one byte per character for basic Latin script, three for Devanagari or CJK under UTF-8), two factors that do not always pull in the same direction. Within the class of systems that segment all languages with a single shared finite inventory and bound computation by segment count, no known alternative eliminates the cross-linguistic disparity; every known alternative relocates it. The specification of that class requires justification, since a constitutivity claim relativised to a class the analyst has drawn is worth no more than the class. C is not stipulated here; it is read off the constraints under which the systems at issue are actually built and sold. Commercial interfaces price access, bound context, and meter latency by segment count, so the segment is the unit in which the disparity is realised as a cost to users; the tooling, evaluation suites, and adaptation pipelines built around a deployed model presuppose a shared inventory, so per-language inventories are not an available substitution within the current ecosystem; and the fixed context windows of current architectures make segment count a binding resource rather than an accounting convention. A class bounded by these constraints is the space in which developers and deployers actually choose, which is what the criterion asks C to be. Should the constraints lapse, C expands and the classification lapses with it, which is what Sect. 4.4 requires a well-formed constitutivity claim to concede. Our classification is therefore the hedged one that Sect. 4.4 mandates: the residual disparity is constitutive relative to that class, on current knowledge, and the classification would be refuted by exhibiting a shared-inventory scheme with cross-linguistically uniform encoding costs, or dissolved by abandoning the shared-inventory architecture itself, which would amount to expanding C.
The case shows the criterion doing work that an undifferentiated vocabulary cannot: neither tokeniser bias is constitutive nor tokeniser bias is contingent is well-formed as it stands; the disparity decomposes, the components classify differently, and the classification of the residual component wears its defeat condition on its face.
5.3 Hiring and clinical risk: entangled layers
The first two cases were chosen because they resolve cleanly. The typical deployed system does not, and the pair of cases in this section, one documented in industry reporting and one in a landmark empirical study, shows the criterion operating where contingent and residual layers are entangled.
Automated hiring activates the full sequence of representational operations in one tightly coupled pipeline, which is why it serves here as the paradigm case, anchored to a documented system rather than to a hypothetical one. Amazon developed and ultimately abandoned an experimental CV-screening system after discovering that it penalised CVs containing indicators of being a woman, including attendance at women’s colleges and the word women’s itself; attempts to neutralise the explicit indicators did not secure the system against learning subtler proxies, and the project was terminated [Dastin (2018); the episode and the design space of automated hiring tools are analysed in the scholarly literature by Raghavan et al. (2020) and Sánchez-Monedero et al. (2020)]. The pipeline can be described as four successive representational moments, which we use as descriptive labels rather than as a formal apparatus:Footnote 5 discretisation, as the narratively structured CV is parsed into fields, and whatever resists tokenisation (an unconventional trajectory, resilience in adversity) is dropped; decontextualisation, as formally identical credentials obtained under profoundly different circumstances (the first-generation graduate who worked through a degree, the legacy student at a well-resourced institution) become computationally indistinguishable; quantification, as the multi-criteria and partly incommensurable judgement of a competent recruiter is collapsed into a score whose constitutive choices are no longer visible in it; and optimisation, as the objective, fit to the features of past successful hires, converts the company’s historical selection patterns from a fact about the past into a criterion for the future.
The criterion sorts the layers. The disparity documented in the Amazon case was contingent along at least one dimension and was treated as such: the training target and cohort were an operationalisation of merit as similarity to a historically male-dominated sample, and admissible alternatives (outcome-based targets, restructured cohorts, structured competency assessments in place of free-text CV mining) exist within the class of automated screening for the same purpose. Whether a residual layer remains is exactly the question the burden structure of Sect. 4.5 assigns: the deployer who claims that no admissible operationalisation of hiring merit avoids reproducing contextual disadvantage owes the documented exhaustion; the critic who claims the whole disparity is removable owes an exhibit that survives evaluation. The Amazon engineers’ failure to secure the system against proxy learning is, in the terms of Sect. 4.6, the failure of particular remedies, evidence about the depth of the disparity, not a proof of constitutivity.
The clinical case supplies what the hiring case cannot: an empirical demonstration that re-operationalisation, the intervention our representational LDA test quantifies over, can remove a disparity that model-level interventions could not reach. Obermeyer et al. (2019) analysed a commercial algorithm applied, at the time, to roughly two hundred million people annually to identify patients for high-risk care management. The algorithm predicted health costs as its operationalisation of health need. Because unequal access depresses spending on Black patients conditional on need, equally sick Black patients received systematically lower scores; at a given score, Black patients carried substantially more chronic illness, and the authors calculated that removing the bias would raise the fraction of Black patients receiving additional help 17.7% to 46.5%. The disparity was invisible to model-level audit, the model predicted its target well and was calibrated in cost, and it was removed, in the authors’ experiments with the manufacturer, by changing the target: re-operationalising need through measures of health rather than through cost alone reduced the disparity by 84%. Relative to the purpose of allocating care management to the sickest patients, the operationalisation of need as cost was a contingent representational choice with a demonstrated less-discriminatory alternative, and the case is the empirical anchor for the level shift defended in Sect. 4.2.
What remains after the correction is the honest residuum, and we state it as Sect. 4.4 requires. The social determinants that shape health trajectories, housing instability, food insecurity, environmental exposure, discrimination within the healthcare system itself, are captured by structured clinical data only through coarse proxies, and a patient’s subjective experience of illness and the biographical meaning of a diagnosis fall below the epistemic grain of any current clinical data model. Whether these limits mark a constitutive layer relative to the class of structured-data clinical prediction, or only the present boundary of an expanding C, is not settled by the framework, and we do not settle it here; the criterion’s contribution is that the question is now well-formed, parameterised, and assigned to the party whose showing would resolve it.
A closing observation on the shape of these results. Only one of the disparities examined here has been classified as constitutive, the residual component of Sect. 5.2, and that classification is relative to a class and hedged to current knowledge. The pattern is not an artefact of case selection. Because C expands with technique, unhedged constitutivity claims are exactly the ones the criterion forbids, and a framework that delivered a catalogue of permanently irremediable disparities would be one that had stopped taking the defeasibility of its own negative existentials seriously. What the distinction contributes is what it requires of the parties before a disparity is accepted as unavoidable, not the length of the list of disparities that turn out to be.
6 The impossibility theorems revisited
The relation between the criterion and the impossibility theorems of algorithmic fairness (Kleinberg et al. 2017; Chouldechova 2017) needs stating precisely, because two claims about it are true at once and are easily run together. The theorems are not consequences of the representational operations analysed in Sects. 2 and 3. They are facts about joint distributions: whenever base rates differ across groups and prediction is imperfect, calibration and balance for the positive and negative classes cannot be jointly satisfied, and this holds however rich, continuous, or fine-grained the representation is [for a critical review of the criteria and of the assumptions governing their comparison, see Corbett-Davies and Goel (2018)]. No refinement of the representation dissolves the incompatibility, and nothing in the framework proposed here should be read as suggesting that the ontological fracture produces it.Footnote 6
That is the negative claim, and it exhausts what the boundary requires. The positive claim is that the criterion of Sect. 4 applies to the theorems exactly, and classifies them. Fix P as the joint satisfaction of calibration and balance for the positive and negative classes; fix D as the shortfall from that joint satisfaction; and let C be the class of designs that hold the partition of the population and the base rates fixed. The theorems establish that no member of that C serves P. This is a constitutivity claim in the sense of Sect. 4.4, and it is the rare one whose negative existential is proved rather than provisionally documented: C is not searched but quantified over. The theorems are therefore not an exception to the criterion but its limiting case, and the one place in this literature where a constitutivity claim is demonstrated rather than defeasibly supported.
What makes that C the class it is, however, is not settled by the mathematics. The results are stated relative to a fixed partition of the population into groups and relative to base rates treated as exogenous parameters, and both are representational commitments of exactly the kind analysed above: the partition is an ex ante precisification of open-textured social categories [Sect. 3.1; on the constituted rather than measured character of the protected categories themselves, see Benthall and Haynes (2019), Hanna et al. (2020) and Hu and Kohler-Hausmann (2020)], and the base rates, in domains such as criminal justice, credit, and employment, are in substantial part the sediment of historically produced inequality, which their encoding as fixed parameters naturalises. The mathematics is silent on which partition matters and on whether the base rates deserve the status of givens. Those are decisions taken before the theorem applies, and they, not the theorem, are where the present analysis has purchase.
The constitutivity the theorems establish is therefore relative to its class in the way every classification under the criterion is, and the class is bounded by choices rather than by nature. Base rates are not primitives of a domain; they are properties of a measured construct, and a change of construct changes them. Recidivism operationalised as re-arrest and recidivism operationalised as reconviction do not have the same base rates, and neither of them is the rate at which offences are committed. Health need operationalised as expenditure and health need operationalised as morbidity do not have the same base rates either, which is what the clinical case of Sect. 5.3 turns on: re-operationalising the target moved the disparity by moving what the base rates were rates of. Admitting re-operationalisations of this kind expands C beyond the class over which the theorems quantify, and an impossibility relative to the narrower class does not travel with the expansion. Nothing guarantees that an admissible re-operationalisation equalising the base rates exists in a given domain, or that one would still serve the purpose by which the deployment is defended; that is precisely the showing the burden structure of Sect. 4.5 assigns, and it is a showing, not a stipulation. What the criterion changes is that the question becomes askable. What is inescapable within C is not thereby inescapable, and marking that difference is what the criterion is for.
Hellman’s (2020) argument that the choice amongst the conflicting criteria encodes a prior conception of the wrong of discrimination completes the picture from the normative side: which horn of the impossibility a deployment accepts is a decision about values, taken under mathematical constraint but not made by it.
The distinction between what a formal result establishes and what it presupposes also clarifies the relation between the present framework and Selbst et al.’s (2019) abstraction traps. The traps, framing, portability, formalism, ripple effects, solutionism, are patterns of failure that arise when technical work abstracts from the social context of deployment; the representational analysis of Sects. 2 and 3 identifies the common mechanism, the ex ante closure of context-sensitive concepts, of which several of the traps are recognisable special cases. We advance this as a unification claim about mechanism, not as a replacement for the traps’ diagnostic vocabulary, which remains finer-grained at the level of practice.
7 Regulatory implications: ontological auditing
7.1 The limits of the current framework
The EU Artificial Intelligence Act (Regulation 2024/1689) adopts a risk-based architecture: Article 6 classifies systems,Footnote 7 with the high-risk category of Annex III covering, amongst others, employment, credit scoring, law enforcement, and migration; high-risk systems are subject to obligations concerning risk management, data governance, transparency, and human oversight. From the perspective developed here, these obligations address predominantly what Sect. 4 classifies as contingent sources of disparity, and they address them at the level of data and model rather than at the level of representation. Article 10 requires training, validation, and testing data to meet quality and representativeness standards, which disciplines the data given the categories but not the choice of the categories; Article 13 requires transparency and instructions for use, which makes the system’s operations and logic visible without reaching the alternative operationalisations that were never built (the residual opacity of Sect. 3.2); Article 14 requires effective human oversight, which positions an overseer downstream of a precisification the overseer cannot re-open (Sect. 3.1). Lanamäki et al. (2025) describe the compliance side of this: under the open texture of the AI Act, organisations must reach closure on what the requirements demand before the interpretive sources that would settle the question are available. The present analysis adds the other side. The closure so reached is then implemented in systems whose representational choices are themselves fixed in advance, so that the regulatory response reproduces, one level up, the ex ante closure the requirements were meant to control.
Municipal and judicial practice show both the reach and the limit of output-level control. New York City Local Law 144 of 2021, in effect from 1 January 2023 and enforced from 5 July 2023, requires annual independent bias audits of automated employment decision tools with public disclosure of results;Footnote 8 the Tribunal of Bologna’s order of 31 December 2020 recognised that a platform’s shift-allocation algorithm produced indirectly discriminatory effects against riders exercising protected rights, absent any discriminatory intent.Footnote 9 Both instruments operate on deployed systems and their outputs; neither reaches the prior question of how the categories, targets, and objectives were chosen, which the Obermeyer case shows can be where the disparity lives and where the remedy lies.
7.2 The ontological impact statement
We propose the practice of ontological auditing: the systematic, upstream evaluation of the representational choices constituting an algorithmic system, conducted during design, before data collection and architecture are finalised. The term is used in a philosophical and normative sense, distinct from the consistency-checking of formal OWL ontologies in knowledge representation research.Footnote 10 Its documentary instrument is the Ontological Impact Statement, and Sect. 4 now allows us to say precisely what the statement is: the standing form of the deployer’s burden of exhaustion under the representational LDA test.
For each socially consequential construct the system operationalises, the statement would document: (a) what phenomenon the variable purports to represent and how it was operationalised; (b) which dimensions of the phenomenon the operationalisation excludes, and with what anticipated effect; (c) what proxy relationships link the chosen variables to protected characteristics; (d) which alternative operationalisations were considered, on what grounds each was excluded from the admissible class or found not to serve the purpose, and what searches were undertaken; and (e) for any disparity the deployer regards as unremediable, the parameters (P, D, C) relative to which that claim is made and the exhaustion supporting it. The clinical algorithm of Sect. 5.3 shows what the entries would contain. Under (a), the variable is projected health care expenditure, offered as an operationalisation of health need. Under (b), the operationalisation excludes illness that does not generate billed encounters, with the anticipated effect of understating need wherever access to care is constrained. Under (c), expenditure stands in a documented proxy relationship to race through that unequal access. Under (d), the alternatives are the measures of health and the combined targets subsequently tested by the authors with the manufacturer, together with the grounds on which the cost target had been preferred, namely its availability in claims data and its performance against its own target. Under (e), no residual constitutivity claim would be sustainable since the alternative was exhibited and substantially reduced the disparity. A statement of this form, filed before deployment, would have exposed the choice that produced the disparity whilst it was still cheap to revise. Requirement (d) is what differentiates the instrument from existing practice. Algorithmic impact assessments, fundamental rights impact assessments, datasheets, and model cards document what a system is and does (Moss et al. 2021; Reisman et al. 2018); none of them requires the deployer to answer, at the representational level, the question disparate-impact doctrine asks at the output level, whether a less-discriminatory way of achieving the same end was available. The Ontological Impact Statement asks exactly that question, one level up, and records the answer in a form that an auditor, a court, or an affected party can contest by exhibit.
Within the AI Act’s architecture, the practice has two natural points of integration: the Article 9 risk management system, which mandates identification and mitigation of risks but does not currently name representational risk as a category, and the Article 10 data governance requirements, which discipline data quality but not the prior choice of the categories structuring the data. Evaluation of the statements would require interdisciplinary capacity, and we retain, as an explicitly untested institutional proposal, the suggestion of sociotechnical review committees including domain experts, social scientists, philosophers of science, legal scholars, engineers, and representatives of affected communities, with a mandate to assess the adequacy and justification of representational choices rather than to approve systems.
We do not claim that ontological auditing eliminates constitutive discrimination; where a disparity is genuinely constitutive relative to a purpose and the available design space, the analysis of Sect. 4 entails that no audit removes it. The claim is procedural: the practice makes representational choices visible and contestable before they harden into infrastructure, converts constitutivity claims from rhetorical shields into documented and defeasible showings, and shifts a portion of regulatory attention from the post hoc correction of outputs to the upstream scrutiny of the choices that produce them.
8 Limitations
Seven limitations of the analysis should be stated plainly.
First, the criterion classifies disparities relative to a purpose, a disparity measure, and an admissible design space, and it offers no theory of how those parameters are themselves to be fixed; that is a normative task the criterion structures but does not perform, and reasonable specifications can classify the same disparity differently.
Second, the classifications offered in Sect. 5 are defeasible, and two of them revise classifications that the representational vocabulary of Sects. 2 and 3, taken alone, would support; readers should expect further revisions as the design space expands, and should treat that openness as a property of the criterion rather than a defect of its applications.
Third, operationalising the representational LDA test inherits the fundamental difficulties documented for its model-level counterpart, including the dependence of any comparison on adequate held-out data (Laufer et al. 2025), compounded by the cost of developing representational alternatives far enough to evaluate them.
Fourth, the representational operations analysed here are not exclusive to computation; they characterise formalised bureaucratic decision-making generally (Bowker and Star 2000; Espeland and Stevens 1998). What the paper claims as specific to the computational case is the conjunction identified in Sect. 3: ex ante and uniform precisification of open-textured concepts, at scale, without an interpreter positioned to re-open the precisification in situ.
Fifth, the case studies are illustrative and adversarially selected to test the criterion, not a representative sample of deployed systems, and the entangled cases of Sect. 5.3 end, deliberately, in assigned burdens rather than settled classifications.
Sixth, a reflexive limit: this paper itself analyses representational practice through a simplified vocabulary, and its own distinctions coarsen the phenomena they organise; we have tried to keep the apparatus no larger than the argument requires, and the criterion of Sect. 4, not the vocabulary of Sect. 2, is the part on which the argument rests.
Seventh, the governance proposal of Sect. 7 is untested, and its documentary instrument overlaps in genre with existing impact assessment practice; its differentiating element, the representational-level alternatives requirement of clause (d), has not been trialled, and its administrative cost is unknown.
9 Conclusion
The fairness of an algorithmic system is decided in significant part before any fairness metric applies. It is decided in the choice of the categories the system recognises, the constructs it operationalises, the features through which it observes its domain, and the objective it optimises. Those choices are made once, at design time; they close ex ante concepts whose institutional use is open-textured; and everything downstream inherits them, including the metrics that are meant to evaluate the system for fairness. An account of algorithmic discrimination that begins at the metric has already conceded the ground on which the question is settled.
This paper has proposed a criterion for deciding when disparities arising from those choices are remediable. A disparity is contingent, relative to a legitimate purpose, a measure of disparity, and a class of admissible operationalisations, when some member of that class supports a system that serves the purpose with less disparity; it is constitutive relative to those parameters when no member does. The criterion is a representational analogue of the less–discriminatory alternative, and three of its features carry the argument. It is relative, so that every classification is elliptical for the parameters that generate it, and disagreements about classifications resolve into separately arguable disagreements about purpose, about the relevant measure of disparity, and about the design space actually available. It is defeasible, so that a constitutivity claim carries its own refutation condition and lapses when the design space expands. And it allocates burdens, so that contingency is shown by exhibit and constitutivity by documented exhaustion, neither of them discharged by assertion.
What this buys is a change of register. Whether a given bias is inherent to computation has the grammar of a metaphysical question and the argumentative behaviour of one: it is contested indefinitely because nothing counts as settling it. Under the criterion the question becomes procedural. The claim that a disparity could not have been avoided is a claim someone must make, relative to parameters they must state, supported by a search they must document, and defeasible by a single exhibit that anyone may produce. The case studies show what this looks like when it bites. A bias whose upstream location made it a natural candidate for inevitability turns out to have been remediated in fact (Sect. 5.1). A disparity that resists classification as a whole decomposes into a component for which alternatives are available and a residual component whose classification wears its defeat condition on its face (Sect. 5.2). A pair of deployed systems ends not in a verdict but in an allocation of burdens, which is the honest outcome where the evidence is not yet in (Sect. 5.3). The impossibility theorems are the limiting case of the same structure: a constitutivity claim that is proved rather than documented, and proved relative to a class of designs holding fixed a partition and a set of base rates that are themselves representational commitments (Sect. 6). Only one of the disparities examined here is classified as constitutive, and that classification is relative to a class and hedged to current knowledge. A framework of this kind is not in the business of producing a catalogue of permanent impossibilities. It is in the business of raising the price of claiming one.
The regulatory consequence is correspondingly concrete. The obligations now in force, in the AI Act and in the audit regimes emerging around it, discipline data quality, model behaviour, and outputs, all of them downstream of the representational choices that fix what the data can record and what the outputs can distinguish. Ontological auditing, with the Ontological Impact Statement as its documentary form, adds one requirement that no existing instrument imposes: that the deployer answer, at the representational level, the question disparate-impact doctrine has long asked at the level of outputs, whether a less-discriminatory way of achieving the same legitimate end was available, and record the answer in a form that an auditor, a court, or an affected party can contest by exhibit. The proposal is untested, its administrative cost is unknown, and the institutional capacity it presupposes has still to be built; Sect. 8 says so rather than claiming otherwise.
Three lines of work follow. The criterion is stated informally here and admits a formal development, as a theory of comparative information loss across admissible design spaces, which would supply necessary conditions for constitutivity that the present formulation can only approach through exhaustion. The delimitation of the admissible class deserves study in its own right, since a class bounded by commercial convention rather than by technical necessity, as in the tokenisation case of Sect. 5.2, yields constitutivity claims whose proper remedy is regulatory rather than technical, and a mature version of the criterion should mark that difference explicitly. And the Statement of Sect. 7.2 needs trialling on real design processes, where the question is not whether the requirement is coherent but whether it is answerable at acceptable cost.
Where a representational choice is genuinely forced, the criterion asks that the fact be established rather than asserted, and weighed openly rather than discovered after deployment. Where it is not forced, and the analysis offered here suggests that this is the more common case, the criterion asks the simpler question that the vocabulary of inevitability has tended to obscure: what else could have been built, and why was it not?
Data availability
No datasets were generated or analysed during the current study.
Notes
Classic examples of contingent disparities at the data and feature level include the accuracy gap of commercial gender classifiers on darker-skinned faces, traceable to unrepresentative training sets and remediable by representative data collection (Buolamwini and Gebru 2018), and postcode variables proxying ethnicity, removable or adjustable once identified. The proxy problem is paradigmatic of contingency that nonetheless survives piecemeal correction, since removing one proxy leaves correlated proxies in place (Dwork et al. 2012). For a survey of the technical intervention landscape, see Mittelstadt et al. (2016).
The definition is stated informally by design. It admits formal development, as a theory of comparative information loss across admissible design spaces, and we pursue that development in separate technical work; nothing in the present argument depends on it. The criterion’s content here is carried entirely by its relativisation to explicit parameters, its falsification conditions (Sect. 4.4), and its burden structure (Sect. 4.5), which is the register of rigour this argument requires.
Earlier work within this framework classified this case as constitutive, on the strength of the upstream location of the bias. The criterion of Sect. 4 reverses that classification, which is part of the reason for presenting the case first.
On the philosophical implications of tokenisation and of the relation between linguistic form and meaning in language models, see Bender and Koller (2020) and Bender et al. (2021). Nothing in the argument of this section turns on how that relation is settled: the disparity analysed here is a property of the segmentation itself, measurable before any question about what the segments mean arises.
The Baconian borrowing is deliberate but heuristic. Just as Bacon’s idola catalogued systematic distortions of human cognition, the idola machinae name systematic distortions introduced by the computational apparatus itself. We make no historiographical claim that Bacon’s fourfold was open to extension; the vocabulary serves as a unifying register for the four moments described here, not as a taxonomic thesis.
Article 6 distinguishes systems that are high-risk as safety components of regulated products (Article 6(1)) from those that are high-risk because they fall within the application areas listed in Annex III (Article 6(2)). The systems discussed in this paper fall predominantly under the second category.
The law is amongst the first attempts in a major Western jurisdiction to impose a transparency-based regulatory regime on automated hiring. From the perspective of the present framework, an audit-and-disclosure regime of this kind addresses primarily contingent forms of algorithmic disparity since it evaluates a deployed tool against its own representational choices rather than against the alternatives to them.
Tribunale di Bologna, sezione lavoro, ordinanza of 31 December 2020, concerning the shift-allocation algorithm of a major food-delivery platform. The Court found indirectly discriminatory effects against riders who exercised the right to strike or to abstain from work for protected reasons, absent discriminatory intent; the decision is widely regarded as a landmark in the European recognition of indirect algorithmic discrimination.
The knowledge representation notion refers to the consistency-checking and quality assessment of formal OWL ontologies; see Gangemi et al. (2006) for a representative example. The two practices share a concern with the adequacy of representational structures but operate at different levels of abstraction and address different audiences.
References
Ahia O, Kumar S, Gonen H, Kasai J, Mortensen DR, Smith NA, Tsvetkov Y (2023) Do all languages cost the same? Tokenization in the era of commercial language models. In: Proceedings of the 2023 conference on empirical methods in natural language processing. Association for Computational Linguistics, pp 9904–9923. https://doi.org/10.18653/v1/2023.emnlp-main.614
Ananny M, Crawford K (2018) Seeing without knowing: limitations of the transparency ideal and its application to algorithmic accountability. New Media Soc 20(3):973–989. https://doi.org/10.1177/1461444816676645
Angwin J, Larson J, Mattu S, Kirchner L (2016) Machine bias. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
Baldini D, De Benedetto M (2025) The open texture of ‘algorithm’ in legal language. AI Soc 40(3):1643–1654. https://doi.org/10.1007/s00146-024-01925-z
Barocas S, Hardt M, Narayanan A (2019) Fairness and machine learning: limitations and opportunities. fairmlbook.org. https://fairmlbook.org
Bender EM, Koller A (2020) Climbing towards NLU: on meaning, form, and understanding in the age of data. In: Proceedings of the 58th annual meeting of the Association for Computational Linguistics, pp 5185–5198. https://doi.org/10.18653/v1/2020.acl-main.463
Bender EM, Gebru T, McMillan-Major A, Shmitchell S (2021) On the dangers of stochastic parrots: can language models be too big? In: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp 610–623. https://doi.org/10.1145/3442188.3445922
Benthall S, Haynes BD (2019) Racial categories in machine learning. In: Proceedings of the conference on fairness, accountability, and transparency, pp 289–298. https://doi.org/10.1145/3287560.3287575
Bowker GC, Star SL (2000) Sorting things out: classification and its consequences. MIT Press
Buolamwini J, Gebru T (2018) Gender shades: intersectional accuracy disparities in commercial gender classification. In: Proceedings of the 1st conference on fairness, accountability and transparency. PMLR, pp 77–91
Burrell J (2016) How the machine ‘thinks’: understanding opacity in machine learning algorithms. Big Data Soc 3(1):1–12. https://doi.org/10.1177/2053951715622512
Cartwright N (1983) How the laws of physics lie. Oxford University Press
Chouldechova A (2017) Fair prediction with disparate impact: a study of bias in recidivism prediction instruments. Big Data 5(2):153–163. https://doi.org/10.1089/big.2016.0047
Corbett-Davies S, Goel S (2018) The measure and mismeasure of fairness: a critical review of fair machine learning. https://doi.org/10.48550/arXiv.1808.00023
Crawford K, Paglen T (2021) Excavating AI: the politics of images in machine learning training sets. AI Soc 36(4):1105–1116. https://doi.org/10.1007/s00146-021-01162-8
Dastin J (2018) Amazon scraps secret AI recruiting tool that showed bias against women. Reuters [reprinted in K. Martin (Ed.), Ethics of data and analytics: concepts and cases (pp. 296–299). Auerbach Publications, 2022]
Di Giorgio A (2026) Quando la macchina sbaglia. Capire e contrastare le discriminazioni algoritmiche. Mimesis
Dwork C, Hardt M, Pitassi T, Reingold O, Zemel R (2012) Fairness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference. ACM, pp 214–226. https://doi.org/10.1145/2090236.2090255
Espeland WN, Stevens ML (1998) Commensuration as a social process. Annu Rev Sociol 24:313–343. https://doi.org/10.1146/annurev.soc.24.1.313
Floridi L (2008) The method of levels of abstraction. Minds Mach 18(3):303–329. https://doi.org/10.1007/s11023-008-9113-7
Friedler SA, Scheidegger C, Venkatasubramanian S (2021) The (im)possibility of fairness: different value systems require different mechanisms for fair decision making. Commun ACM 64(4):136–143. https://doi.org/10.1145/3433949
Gangemi A, Catenacci C, Ciaramita M, Lehmann J (2006) Modelling ontology evaluation and validation. In: Sure Y, Domingue J (eds) The semantic web: research and applications (ESWC 2006), vol 4011. Springer, pp 140–154. https://doi.org/10.1007/11762256_13
Hacking I (1999) The social construction of what? Harvard University Press
Hacking I (2007) Kinds of people: moving targets. Proc Br Acad 151:285–318
Hanna A, Denton E, Smart A, Smith-Loud J (2020) Towards a critical race methodology in algorithmic fairness. In: Proceedings of the 2020 conference on fairness, accountability, and transparency, pp 501–512. https://doi.org/10.1145/3351095.3372826
Hart HLA (1961) The concept of law. Oxford University Press
Hedden B (2021) On statistical criteria of algorithmic fairness. Philos Public Aff 49(2):209–231. https://doi.org/10.1111/papa.12189
Hellman D (2020) Measuring algorithmic fairness. Va Law Rev 106(4):811–866
Hu L, Kohler-Hausmann I (2020) What’s sex got to do with machine learning? In: Proceedings of the 2020 conference on fairness, accountability, and transparency, p 513. https://doi.org/10.1145/3351095.3375674
Jacobs AZ, Wallach H (2021) Measurement and fairness. In: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp 375–385. https://doi.org/10.1145/3442188.3445901
Kleinberg J, Mullainathan S, Raghavan M (2017) Inherent trade-offs in the fair determination of risk scores. In: Proceedings of the 8th innovations in theoretical computer science conference, vol 67, pp 43:1–43:23. https://doi.org/10.4230/LIPIcs.ITCS.2017.43
Lanamäki A, Viljanen M, Väyrynen K, Bennett Moses L (2025) Legal compliance and the open texture of law. J Assoc Inf Syst 26(1):1–8. https://doi.org/10.17705/1jais.00922
Laufer B, Raghavan M, Barocas S (2025) What constitutes a less discriminatory algorithm? In: Proceedings of the 2025 symposium on computer science and law. ACM, pp 136–151. https://doi.org/10.1145/3709025.3712214
Mittelstadt BD, Allo P, Taddeo M, Wachter S, Floridi L (2016) The ethics of algorithms: mapping the debate. Big Data Soc 3(2):1–21. https://doi.org/10.1177/2053951716679679
Morrison M, Morgan MS (1999) Models as mediating instruments. In: Morgan MS, Morrison M (eds) Models as mediators: perspectives on natural and social science. Cambridge University Press, pp 10–37
Moss E, Watkins EA, Singh R, Elish MC, Metcalf J (2021) Assembling accountability: algorithmic impact assessment for the public interest. Data and Society Research Institute. https://datasociety.net/library/assembling-accountability-algorithmic-impact-assessment-for-the-public-interest/
Obermeyer Z, Powers B, Vogeli C, Mullainathan S (2019) Dissecting racial bias in an algorithm used to manage the health of populations. Science 366(6464):447–453. https://doi.org/10.1126/science.aax2342
Pasquale F (2015) The black box society: the secret algorithms that control money and information. Harvard University Press
Petrov A, La Malfa E, Torr PHS, Bibi A (2023) Language model tokenizers introduce unfairness between languages. In: Advances in neural information processing systems, NeurIPS 2023, vol 36
Potochnik A (2017) Idealization and the aims of science. University of Chicago Press
Quaresmini C, Primiero G (2024) Data quality dimensions for fair AI. In: Proceedings of the 2nd workshop on fairness and bias in AI co-located with ECAI 2024 (CEUR Workshop Proceedings), vol 3808. https://ceur-ws.org/Vol-3808/
Raghavan M, Barocas S, Kleinberg J, Levy K (2020) Mitigating bias in algorithmic hiring: evaluating claims and practices. In: Proceedings of the 2020 conference on fairness, accountability, and transparency. ACM, pp 469–481. https://doi.org/10.1145/3351095.3372828
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L series
Reisman D, Schultz J, Crawford K, Whittaker M (2019) Algorithmic impact assessments: a practical framework for public agency accountability. AI Now Institute. https://ainowinstitute.org/wp-content/uploads/2023/04/aiareport2018.pdf
Sánchez-Monedero J, Dencik L, Edwards L (2020) What does it mean to ‘solve’ the problem of discrimination in hiring? Social, technical and legal perspectives from the UK on automated hiring systems. In: Proceedings of the 2020 conference on fairness, accountability, and transparency. ACM, pp 458–468. https://doi.org/10.1145/3351095.3372849
Selbst AD, Boyd D, Friedler SA, Venkatasubramanian S, Vertesi J (2019) Fairness and abstraction in sociotechnical systems. In: Proceedings of the conference on fairness, accountability, and transparency, pp 59–68. https://doi.org/10.1145/3287560.3287598
Tschider CA (2021) Legal opacity: artificial intelligence’s sticky wicket. Iowa L Rev Online 106:126–136
Waismann F (1945) Verifiability. Proc Aristot Soc Suppl 19:119–150
Weisberg M (2013) Simulation and similarity: using models to understand the world. Oxford University Press
Yang K, Qinami K, Fei-Fei L, Deng J, Russakovsky O (2020) Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy. In: Proceedings of the 2020 conference on fairness, accountability, and transparency, pp 547–558. https://doi.org/10.1145/3351095.3375709
Funding
Open access funding provided by Università del Salento within the CRUI-CARE Agreement.
Author information
Authors and Affiliations
Contributions
A.D.G. is the sole author of the manuscript. A.D.G. conceived the study, conducted the research, wrote the manuscript, and reviewed the final version.
Corresponding author
Ethics declarations
Conflict of interest
The author declares no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Di Giorgio, A. When algorithms cannot be fair: constitutive and contingent discrimination and the limits of less-discriminatory alternatives. AI & Soc (2026). https://doi.org/10.1007/s00146-026-03384-0
Received:
Accepted:
Published:
Version of record:
DOI: https://doi.org/10.1007/s00146-026-03384-0


