On Under-Determinism
— or why a cowboy must get off the horse, and talk.
For some time now, I've been feeling there is a quite a lot of difference between statistical way of thinking about learning vs the actual practice of machine learning. It has been to the point that I think statistical way is misleading and gives a wrong picture of the field. This note is an attempt to verbalise this feeling rather properly. I'll also talk about randomness, and aleatoric and epistemic dichotomy, and an argument why we need to abandon this terminology (or rather abandon the source of this dichotomy), and replace it what I call under-determinism. Finally, I'll lift my argument (or rather it'll naturally be lifted) to the broader AI alignment and ethics discourse.
Let's start with the definitions first, and I'll distinguish between statistical way of thinking about learning and actual machine learning way of thinking about learning. To illustrate what I mean by learning, consider the simple task of predicting the risk of a certain disease in individuals from certain population. Given an individual $\omega$ from this population, we want to assign the level of risk this person has for this disease. Now as a very crude predictor, we can always assign the marginal rate of this disease in this population as a risk estimate for this individual. That is I go out in this population and record some past occurrence of this disease. We call this the outcome $y \in \{0,1\}$. And say for $\omega$, we can assign a certain $y$, and hence we have some information of the form $\{(\omega_i, y_i)\}_{i=1}^{N}$. However, say I want to give a rather tailored estimate of the risk for this individual. To do so, I'll start collecting certain features, that I think might be relevant for the task at hand, like maybe age, gender, demographic, previous health status, etc. We call this, the feature representation of the individual, denoted as $x$. Then the learning problem becomes predicting the outcome $y$ based on the features $x$. Now obviously, I have belaboured the setup here. But now here we can highlight an operational difference between statistical way of thinking about learning and actual machine learning.
Given the features $x \in \mathcal{X}$ and the outcomes $y \in \{0,1\}$ with denoting $\mathcal{Z} = \mathcal{X} \times \{0,1\}$, one considers the aggregate statistics (or a snapshot so to say) of the features and the outcomes, and call it the distribution $\mathcal{D}$ on $\mathcal{Z}$. And statistical way of thinking about learning aims to estimate certain property of this snapshot, which in this case would be $h^{*}\left(x\right):=\mathbb{P}_{\mathcal{D}}[Y \mid X=x]$, i.e. given the fixed snapshot, what is the best estimate of the outcome $y$ given $x$. $h^{*}(x)$ is also referred to as the Bayes optimal predictor. As a summary, given the fixed snapshot, the learning problem becomes estimating a certain property of it, and since the snapshot is fixed, one can also write down a notion of optimal predictor there. I can now contrast this with what I refer to as the actual machine learning. Note the task is given an individual $\omega \in \Omega$, we have to predict the outcome $y$. I'll say it again: given the individual $\omega$, the task is to predict outcome $y$. And now I think it clarifies: a problem of learning as the way machine learning works operationally would only stop when one can predict the outcome $y$ for the individuals $\omega$ to a certain satisfactorily level. What is the satisfactory level: anything as measured by accuracy, or some other metric of how well the prediction matches the outcome $y$. And to be able to get as perfect accuracy as possible, what one would do is anything that gets the accuracy to go up. This involves collecting better features, baking relevant inductive biases into the predictive model, or any $n$ number of engineering techniques. And this clarifies my main point: a statistical way of thinking about learning started with a fixed snapshot, a certain features and outcome pairs, and this fixed snapshot also defines a ceiling (a Bayes optimal predictor as a property of this snapshot). Actual machine learning, however, only concerns with predicting as best as one can, and to do so, it doesn't operate under the limits of the snapshot: one can literally change the snapshot if that gets the numbers to go up.
To reiterate a bit, the argument is simply this: the actual practice of machine learning cannot be faithfully thought of as estimating the distribution. It is, first and foremost, driven by predicting correctly. Estimating the distribution (or some property of it), or inference, is very much the realm of statistics. A regular problem in statistics deals with something like: given a distribution $\mathcal{N}(\mu, \sigma^{2})$ and certain samples from it $X_1, X_2, \ldots$, what can we say about the parameters $\mu$ and $\sigma^{2}$. But a real-world out there does not have this clean picture of the data coming from a certain distribution that you can infer the defining properties of that distribution. "Distributions don't exist out there, they are constructed."Fortunately, I'm not the only one to say this. Ben Recht, Ben Höltgen, and Bob Williamson have somewhat similar position. As an example, return to the disease prediction case. Imagine $x$ stands for gender, i.e. one has collected a paired dataset of gender-outcome, and now would work under this snapshot. This is now a statistics problem, and the statistics can also provide an optimal solution: look at the gender at give the average outcome rate for this gender as the prediction. But is the machine learning going to be happy with it? Not necessarily. If this "optimal" predictor is not highly accurate, a machine learning problem hasn't been solved. And hence if the gender does not have a good predictive power for the outcome, a machine learning person can collect more features, and hence operate under a completely different distribution over the features and outcomes, and will keep on doing so (and other things) until the prediction performance (accuracy) is sufficiently high.
It might certainly look like I'm attacking a strawman, but I'll first talk some consequences of my arguments. The main consequence is the operational: a closed-world viewpoint vs an open one. I already mentioned the notion of a so-called Bayes optimal one, a ceiling that statistical view offers no longer exists. There is no limit to learning. A consequential extension is that learning, by itself, cannot be capturing the snapshot of the world (an inference argument), as the world is in a constant state of flux. I wouldn't call inferring something about the distribution genuine learning, as if one is able to do so perfectly, then one argues that there is nothing else left to learn. But that cannot be true. If you're not willing to buy this bit of philosophy, my simple argument is that there is no such thing as Bayes optimal. It originates from the closed-world viewpoint, and the practice of machine learning does not operate in that realm. A natural extension of this argument is for the epistemic-aleatoric dichotomy in uncertainty quantification, as I also write here. I won't elaborate much on it here, but I'll say this dichotomy is a direct descendent of the closed-world view, and unfortunately found its way into machine learning due to statistical way of thinking about learning.
Now for the strawman argument. To be clear about what I'm not claiming: I'm not saying the statistical formalism has an error in it. A statistician would rightly point out that the Bayes optimal predictor was always relative to the features, i.e. $h^{*}(x)$ is optimal given $x$, and nothing in the formalism forbids one from collecting more features and conditioning on those too. And that is all fine. I'm mostly talking of the discipline, in two ways of thinking about learning, and of what each way of thinking makes one do. Statistical way of thinking starts from the snapshot and works within it, and everything downstream of it (the optimality, the ceiling, the uncertainty accounting) stays fixed because the snapshot is fixed. The machine learning way of thinking, however, treats the snapshot itself as one more movable part. So yes, a ceiling exists, per snapshot. But a ceiling that moves every time one changes the snapshot is not a ceiling in any operational sense, it is rather a piece of instrumentation so to say: you build it, you use it, and you throw it away when it stops being useful.
And the evaluation, I'd say, is no exception to this. Sure, once I propose a benchmark, that benchmark is a fixed snapshot again, and the performance on it is an ordinary statistical estimate, with all the usual notions applying within it. But consider what happens when the systems get so good at a benchmark that everyone passes it. Does the field declare that we have now measured the quantity we wanted, and we are done? Not really. What the field says instead is that this benchmark has stopped being useful, i.e. it can no longer tell a better system from a worse one, and one simply goes out and proposes a harder benchmark. And this is where all the arguments of falsification come in, as this is almost exactly Popper's picture of science: a model is like a conjecture, a benchmark is like an experiment, and passing an experiment doesn't verify the model, it only means the model has survived so far, and the response to a survival is to propose a harder experiment. So within any single benchmark, sure, the statistics governs the inference. But nothing of this sort governs the proposing of the benchmarks themselves, and there is no ceiling to it either. And hence my position is simply this: the snapshot exists, but only per test, and what does not exist is a final test.
Now the determinism question. I've verbalised this argument on multiple occasions over the last year, and one question that has come every single time is: are you saying the world is deterministic? And my answer is: I don't know, and neither do you, and that is the entire point. I'm not denying that there is irreducible randomness out there, quantum or otherwise. What I'm saying is that we don't know where that lies, and we have no way of locating that boundary from inside our models and our tests. So the best we can do is keep on trying. And while it looks like the inherent randomness in the world would weaken my argument, I tend to think it strengthens it, because a mathematical theory of randomness is a very relativized notion. This is the theory of algorithmic randomness, and its philosophy is exactly the one I'm after: it never asserts anything. It simply says, if my test is failing to detect something, that's on my test. A test that fails to detect a structure in a sequence doesn't certify the sequence as intrinsically random, it only says that this test could not find one, and hence "random", in this theory, always means random relative to these tests. Sure, that could be the limit. But we have no clue, and this is still a very open-minded perspective. And notice this is the falsification argument all over again: nothing is ever verified as random, some things have merely survived our tests so far.
And this is under-determinism, so let me finally define it. Under-determinism is the position that the boundary between what we have not yet resolved and what cannot be resolved is not something one can observe from within any modeling framework, and hence no residual uncertainty ever gets a permanent verdict of being irreducible. The strongest verdict one can ever give is: unresolved by all the tests posed so far. And this is also, at last, my full argument against the aleatoric-epistemic dichotomy. The dichotomy pretends that this boundary is a property of the world out there, something one can point at and say: this part is the noise, stop trying. But that is the closed-world viewpoint speaking again, and it is very much the Bayes optimal ceiling all over again, just in a different form. And this is exactly my argument for adopting the terminology of under-determinism: it is an openness towards the world. Calling an uncertainty under-determined doesn't signal a defeat, the way calling it aleatoric does, but rather a choice, that one has decided to keep on doing something about it. And hence it captures a very faithful view of the limits of learning, without the defeatist closed-world view: the limit is always the current tests, and the current tests can always be improved.
Now I've painted a rather too positive of a picture of the actual practice of machine learning: there are no limits to learning, and machine learning operates scientifically. If one is able to find an error, then one would fix it. Tests are proposed to see how the system is working, and the progress comes from iterating over those tests to pass them, and then proposing the harsher ones. But obviously there is a limit to this viewpoint, and I think this is quite a bit of a frontier challenge. To verbalise it, note that this whole loop of proposing tests and passing them can only run if, at every step, there is a verdict to be had: a test is, after all, a way of producing correct-incorrect judgments, cheaply and repeatably so to say. And hence the entire picture I have painted, the cowboyI've borrowed the cowboy terminology from Tim Van Erven, from the machine learning theory course at the UvA that he co-teaches. mentality of pose a test and do anything to pass it, silently rests on one assumption: that there is a fact of the matter about passing, i.e. there is something out there that can hand us the verdict. Now to see where this assumption breaks, I would like to digress a bit and talk about what is science and what it is not. And now it is time for a history lesson.
I have equated the practice of machine learning with Popper's view of science, but the historical caveat here is that Popper's view came very late. Back in the days of the Enlightenment, when there was an extensive focus on reason, and so much enthusiasm to understand and overcome so much of humanity's problems via it, the picture of what science even means was a completely different one. Science, for them, meant the absolute truth: something demonstrated with certainty, once and for all—I think, therefore I am. And by that standard, they would not even consider physics and chemistry, i.e. what Popper calls science, as science, because being empirical disciplines, these are always at the mercy of the next observation, and hence can never reach that certainty. And now here is the interesting bit: values, morality, and politics enjoyed more of a scientific status than physics and chemistry did, as a great deal of prominent figures genuinely thought these could be settled via pure reason, once and for all. Spinoza, for instance, literally writes his Ethics in the style of Euclid, with definitions, axioms, propositions, etc., and Locke claims that morality can be demonstrated just like mathematics. So the hierarchy, back then, was quite the inverse of ours: virtue was a science, and physics was not quite one.
Clearly, they learned their lesson, though it took a couple of centuries. It turned out the empirical disciplines, the ones always at the mercy of the next observation, are the ones that actually deliver, and they deliver precisely because of this exposure: any error can be found via an observation, and then one would fix it, and this is how the knowledge grows. And so slowly the meaning of the word science moved over to them, and Popper's falsification is the late formalisation of this lesson so to say: a science is not what is certain, a science is what can be refuted. So when I say machine learning operates as a Popperian science, I'm really saying it operates as the science after this lesson was learned. But note the lesson had another half. The project of settling values, morality, and politics via pure reason collapsed, and they could not move over to the new meaning of science either, as there is no test to expose them to, and no observation for them to be at the mercy of. So what happened to them? I think Jürgen Habermas articulates it rather well in his work: Between Facts and Norms. And his two ways of reasoning: the instrumental and the communicative.
Consider first a claim of truth, say, that our predictor achieves such and such accuracy. Such a claim is redeemed against the world: you pose the test, and the world answers, whether you like the answer or not. This is the instrumental reasoning, and it is exactly the picture of machine learning I have painted so far: pose a test and work on it, and there is always a harsher test to pose. Now consider a claim of rightness instead, say, what we ought to value, or what counts as fair, or what counts as harmful. Such a claim is not redeemed against the world, for the simple reason that the world has no opinion on it. It is redeemed only in discourse, through an agreement among those affected by it. This is the communicative reasoning, and here the frame becomes a loose one in a very specific way: there is no test one can pose to the world and get a verdict back. One can still insist on some notion of correct and incorrect here, but I'd say the notion changes its type: a correctness here is a status conferred by an agreement reached under fair conditions, rather than a property measured against something out there. And hence one cannot really benchmark it: it is not that the claim is a fuzzy one, it is that there is no world on the other side of the benchmark to hand us the verdict.
And I don't think this is merely a philosopher's worry, as one can already see some early hints of it in the current practice. Take the alignment of language models via human feedback. Operationally, this is our loop again: pose a test (do the humans prefer this output), and do anything that gets the numbers to go up. But note the target here, the human preference, is not something the world can answer, and it sits much closer to the rightness side of things. And the numbers did go up, but what also came along are the familiar stories of sycophancy and reward hacking: the system learning to optimise the approval so to say, rather than whatever the approval was supposed to stand for. I don't want to over-read these episodes, but they do have the flavour of what one would expect when the instrumental reasoning is pointed at a communicative target. And Habermas, in fact, has a terminology for exactly this kind of failure: the colonization of the lifeworld, the instrumental logic invading a domain that runs on the communicative one. A similar flavour appears in a smaller example: when the human raters disagree about whether an output is harmful, the operational reflex is to average over them, i.e. to treat the disagreement as a noise in the labels. But if what I have said above is right, a disagreement here is not really a noise to be averaged away, and it is rather the very thing to be worked through, via discourse and deliberation.
And now I can say what I find to be the most rewarding bit of this whole exercise: a certain symmetry between the two halves of this note. On the side of the facts, under-determinism refuses to stamp a residual uncertainty as an irreducible noise: the strongest verdict available is that it is unresolved by the tests posed so far. And on the side of the norms, the very same refusal applies: a residual disagreement is not to be stamped as a noise either. So the rule is one and the same on both sides: one should not sign the certificate that closes the world. What differs is only what one does next. On the side of the facts, the refusal sends us back to build a better test. And on the side of the norms, there is no test to build, so the refusal sends us to the discourse, i.e. to talk it through with the ones affected. And this, I tend to think, is why under-determinism is the right terminology to adopt: it doesn't only work where the aleatoric-epistemic vocabulary used to work, it carries over as it is to the very place where the benchmark logic stops working.
So, to close. The statistical way of thinking about learning gave us a closed world: a fixed snapshot, a ceiling, and a category of uncertainty about which nothing further can be done. The actual practice of machine learning recognizes none of these, and I've argued it is right not to: there is no limit to learning to be found in the distribution, the ceiling was always an artifact of the instrumentation, and the honest verdict on any residual uncertainty is only that it is unresolved by the tests posed so far. But there is a limit, and it is of a different kind altogether. It arrives at the point where the next test cannot be posed, but has to be agreed upon. Up to that point, the cowboy can do everything from the saddle. And at that point, the cowboy must get off the horse, and talk.