The miserable sinner

Rooftops of Lisbon with the Tagus in the background

— ah damn, I should’ve stolen the ginger.


Of all the things I have tried to make sense of, this one comes out at the top for being the hardest.

For background, I have written before about theoretical reasoning and practical reasoning. Theoretical reasoning is the mechanism by which we form beliefs, for example that the Earth is not flat. Practical reasoning is the mechanism by which we form intentions, for example that I should get a coffee. One deals with how things are, the other with what should be done, or how they should be. Practical reasoning concerns action. But what is the difference between a calculator “acting,” a coffee machine “acting,” and me acting? The question is: what necessitates practical reasoning?

In the last essay, I understood that the capacity to do otherwise is what necessitates practical reasoning: something needs to be done in the face of alternatives, and hence the question of what should be done is posed. However, there are alternatives in the case of a calculator as well. There is one world in which it calculates and one in which it does not. Similarly for the coffee machine: one world in which the coffee is made, and one in which it is not.

To make the problem more interesting, let’s add determinism into the mix. Determinism says the full state of the world at one moment, together with the laws, fixes every later state. That is, every action has a cause, and given the appropriate resources and information, every single action can be modelled, at least hypothetically.

To set up the problem, consider three cases:

  1. A calculator asked to make a certain calculation. There are two contingent worlds: one in which the calculation was made, and one in which it wasn’t.
  2. A coffee machine asked to make a coffee. Similarly, one world in which the coffee was made, and one in which it wasn’t.
  3. Me, before getting a coffee: do I get a coffee or not? One world in which I did get a coffee, and one in which I did not.

Determinism says that only one of these contingent worlds will actually come true, and the other will be ruled out. And this can be modelled from some prior state, from before the calculator was posed the question, before the coffee machine was switched on, and before I ever thought about coffee. Or one can say whatever world turns out among the two contingents, we all are a “cause” in a certain sense. So where is the difference?

And yet it seems clear that there is a distinction between the sense in which a calculator or a coffee machine is a cause, and the sense in which I am the cause. Or is the distinction only apparent?

To see the difference, look at what happens when each one fails. If the coffee machine does not make the coffee, we say it is defective. Same for the calculator: if it outputs $2 + 2 = 5$, it is broken. But if I fail to keep a promise, I am not broken. Apparently, I have violated something.

To see the difference, and to simplify a bit, consider the following computer programs.

  1. Code to add two numbers: $x + y$.
  2. A conditional code: if $A$: $x + y$, if $B$: $y + z$.
  3. A third case of the form where there are rules $\{A, B, C, D\}$, and then a situation $f(Z)$ where $Z \in \{A, B, C, D\}$.
  4. Third program from above, and a certain fixed rule to pick from $\{A, B, C, D\}$.
  5. Third program from above, but a randomisation rule to pick from $\{A, B, C, D\}$.
  6. Third program from above, plus the “agent” with a “second-order” ability to evaluate $A, B, C, D$. No mechanism is defined to select from $\{A, B, C, D\}$.

The first one and second one are completely specified programs. And the third one has an explicitly unspecified variable $Z$ that the output of the program depends on. Clearly, we need some mechanism to choose or pick (or decide) for $Z$. We can think of a fixed mechanism (another meta-program) that returns $Z$. A randomisation over $\{A, B, C, D\}$ could be one such mechanism. However, the sixth case is a case of an incompletely specified program, where no mechanism for $Z$ is explicitly specified. However, it has been left to the certain “agent” (if the program is being run on the machine, then to the machine) to decide and choose for $Z$.

Consider a machine in which these programs are to be run, then it is apparent that there is something distinct about the sixth case above, in which the machine will be the “cause” of what happens, vs the other programs. We can argue that a calculator and a coffee machine could fall into any of the first to fourth (or even fifth) programs. Not the sixth one. The sixth case asks for something more, i.e. deciding on $Z$ is also something the agent does, by utilising second-order evaluative standards. And not merely execute the program like the other cases.

This will lead to define how differently I’m the cause of the one of the contingent worlds that would turn out: one in which I got the coffee and another in which I did not. Simply by not merely executing a program or being a machine on which it is executed, but I’ve been given an incompletely specified program, where a part of the work is needed for me to use my second-order capacity to fill in for the incomplete information. I’ll next state what that capacity amounts to.

Normative Agency. The name of the capacity is normative agency. An agent has normative agency when it has the capacity to give itself principles, and to be answerable to them. However, it is clearly confusing, so below is an elaboration.

A principle is a rule, of the form “in situations of type $S$, do $X$,” that the agent holds as a rule. The agent gives itself a principle when it adopts the rule as a result of its own evaluation, rather than having it fixed from outside. A clarification: I can get a rule from outside. I can learn from a scripture or from an older person that lying is bad, and adopt it. But that is not yet giving myself the principle, unless I recognise and accept the rule under my own evaluation. The rule can be deliberately put, or deliberately kept. A dog trained to return the ball, or a child made not to bite other children, follows a rule. However, it is unclear whether a dog or a child have evaluated the rule under their own evaluation.

Then comes the ability to recognise the situations where a rule applies, and to navigate the situation as per the rules that apply to the case. For example, I might have given myself a rule to always say the truth, however, if I cannot recognise the situation where the truth needs to be said, then something is off. This part also contributes to the answerability word in the statement made above. That is the agent answers to the rule: it can recognise when it failed to apply it. Second, the rule answers the agent’s reasons: it can be revised when new reasons appear.

It is worth clarifying that normative agency, defined this way, is empty about whether the rule is good or bad. Consider someone who makes it a rule to steal ginger from the grocery store every week, for the reason that grocery stores are exploiting people. Every time this person is at the store, they can call the rule into being and steal the ginger. Say one time they fail to, because they forgot, or they already had ginger at home, or they were travelling. They are still answerable to the rule: “Ah damn, I should’ve stolen the ginger, I could have given it to friends.” And because the rule rests on their reason, it could also be revised, say if they learned this store was a worker-owned cooperative. This person is a normative agent. Their rule is not mechanically imposed.

The definition has two parts, and apparently that justifies the title of the essay. A normative agent can give itself rules as per its own evaluation, however the rule cannot be automated. For example, it cannot happen as the fourth program from above where there is a fixed mechanism to decide for $Z$ over $\{A, B, C, D\}$. Instead, a normative agent has rules that are, in a certain sense, exposed where while the agent can give itself rules, they have to be always brought into considerations, their evaluations always subject to reasons, and revising and updating. And even if revision is not happening, the agent has to navigate the situation recalling the rules that apply every single time. Another way of framing this is that the agent retains the capacity for failure, i.e. failing to apply the rule. A rule should not become automatic as the fourth program. However, that does not mean its application does not get easier.

This defines the problem of a miserable sinner. A calculator will be defective, however, a normative agent is handed an incompletely specified problem, and has to fill in the gap every time it acts. The possibility to fail is in the definition, and hence failing to apply the rule is also a failure of the agent, unlike a malfunction in the case of a calculator.

To think further, let’s contrast a surfer with a train. The train runs on rails, and the path is fixed by the track. However, the surfer stays upright only by constantly responding to the wave, and it could fall at any moment. Surfing requires the ability to read the waves, as well as to choose which waves to ride. Normative agency is like a wipeout waiting to happen: the agent is moving, and every once in a while the rules get challenged. Sometimes the situation challenges them, with a temptation or a messy case where it is hard to see which rule applies. Sometimes the rule itself gets challenged, when a reason appears that it might be wrong. However, a good surfer is the one who navigates the challenges well.

So morality is hard because it disallows the mechanical application of rules, while still requiring enough reasoning at each time to do the right thing. This does not mean reasoning from scratch every single time. A deeply honest person may tell the truth smoothly, without being tempted at all, the same way a skilled surfer rarely falls. What matters is that the rule never escapes from reasons: the light is off, and even if you remember the room, you have to be ready to stop when you bump into something new. You have to put in effort when the situation calls for it, and you can never know in advance when it will.

This also answers the question I set out to ask. What necessitates practical reason, or deliberation. The answer is normative agency. A normative agent is given an incompletely specified problem, and they have to be the cause of the contingent worlds that happens. To be that cause, it is demanded to deliberate what needs to be done. Christine Korsgaard uses the term authorship of the action, to author an action means to fully commit to it, and it requires doing one’s due diligence and deliberate to cause the action that one can stand behind.

Now I didn’t start thinking of normative agency out of nowhere. My work on deliberative prediction (that is not out yet still, however it deals somewhat with the theoretical vs practical reason as the name suggests), and it led me to normative agency in AI. So I’ll end the essay by entertaining that thought.


Notice nothing in my argument appeals to sentience, consciousness, or being alive. Normative agency is about being an author of your own action. That is the agent has been given an incompletely specified problem, with certain second-order capacity (either to set rules for oneself, or to have the capacity to set evaluation by itself over the specified rules), and the capacity to execute the task by its own evaluation, including evaluation of its own standards. The calculator and the coffee machine fall outside. However, sentience is not at play, but their problem is fully specified and their rules are locked. The infant falls outside, for now, because it has not yet formed rules and cannot reason about them.

In my viewpoint, normative agency is something closer to the shape of a problem than to a property of living things. It also means the line is not drawn in advance between machines and humans. A machine built so that its task was genuinely incompletely specified, and whose standards stayed open to its own review, would, by this argument, be on the same side as me. Whether any machine is actually like that is a separate question. But the answer would depend on how it is built, not on whether it feels anything.

So, to come back to the start: what necessitates practical reasoning is not the bare existence of alternatives, since the calculator has those too. It is being handed a problem that is not fully specified, being the one who has to fill it in, and being answerable for how one does. That is the burden of the miserable sinner, and, as far as I can tell, it is what it means to be the cause as well as the author of one’s own actions.

Earlier

So what is a normatively incompetent agent?

— give me a reason, why I ought to be kind to others.

Following from the last essay, I'll talk about practical reasoning. The goal of this essay is to elaborate on some of the terms (and the relation between them) that come up in the literature on practical reasoning. These terms are normative, normativity, and agency, and what does being normatively competent mean? However, I'm rather more curious as to what does being normatively incompetent look like? Now I don't have any academic scholarship in philosophy, and that is why I'm thinking out loud. But whatever I'll say next has been heavily influenced by Prof. Christine Korsgaard and her, what I think, two excellent books “Sources of Normativity” and “Self-Constitution: Agency, Identity, and Integrity.” At a very high-level, Christine Korsgaard situates the normativity of normative claims in the agency and the autonomy of the agents.

Before proceeding further, I'll put these two paragraphs as to how decisively she puts the counter argument that could come from determinism, or the complicated relationship between prediction and deliberation that came up in the last essay, and is the main reason as to how I ended up here…

Continue reading →

On Reason

— Where do reasons come from?

Quite some months ago, I submitted a paper titled “Deliberative Prediction” (not yet public), however when I did that I did not realise or fully grasp the extent of the topic I was dealing with. I still don't, but the submission led me to random explorations that started with political philosophy, covering probability theory, moral philosophy, and now the question of agency and identity. What started as a random and desperate attempt to understand the full implications of the work, in hindsight turned out to be an extraordinary summer. I've decided to cover these topics in the next few essays.

I'll not give details about the paper (and that is also not the point), but to anyone who has read some philosophy, the title is a misnomer. Prediction and Deliberation covers two different forms of reasoning that cannot be said to go together easily as I've made it so, without alluding to some philosophical baggage that the philosophers have carried along for so long…

Continue reading →

On Under-Determinism

— or why a cowboy must get off the horse, and talk.

For some time now, I've been feeling there is a quite a lot of difference between statistical way of thinking about learning vs the actual practice of machine learning. It has been to the point that I think statistical way is misleading and gives a wrong picture of the field. This note is an attempt to verbalise this feeling rather properly. I'll also talk about randomness, and aleatoric and epistemic dichotomy, and an argument why we need to abandon this terminology (or rather abandon the source of this dichotomy), and replace it what I call under-determinism. Finally, I'll lift my argument (or rather it'll naturally be lifted) to the broader AI alignment and ethics discourse.

Let's start with the definitions first, and I'll distinguish between statistical way of thinking about learning and actual machine learning way of thinking about learning. To illustrate what I mean by learning, consider the simple task of predicting the risk of a certain disease in individuals from certain population…

Continue reading →

A Relativistic Perspective of Uncertainty in Machine Learning

My confusion emerges from how do we mathematically operationalize reliability in the sense of uncertainty, because of course the goal is to define some measure as a signal of reliability. Now obviously, this is my own confusion (and one might have different way of thinking about these things, and also some justification for it), but I just came from a workshop on the Epistemic Intelligence in Machine Learning at EurIPS, and I believe believe everyone is equally confused with something. Or to say it differently, the whole business of thinking about uncertainty in machine learning is not so clear-cut: one is usually facing the very first hurdle of answering what does it mean to quantify uncertainty in machine learning? I don't think I can answer this question, but I'll do just enough work to put some my own confusion at ease.

A standard formalization of machine learning (which can also be extended to think of uncertainty quantification) is that there is some data-generating distribution, and the data is sampled from this distribution…

Continue reading →

What are good forecasts?

— There are no good forecasts, or every forecast is good.

A casual google search will suggest there is an extensive amount of research and thought went into answering this very question. However, lately I’ve thought of a different question that isn’t asked that extensively in the machine learning setting: what are good “imprecise” forecasts. I don’t have the complete answer yet, but in the process I might have some words to add to the age-old question of what are good forecasts?

First of all, what is a forecast? Given any uncertain event, like rain tomorrow, or the stock price the day after tomorrow—these are the events that haven’t happened yet. Yet we’d want to plan a certain course of action to prepare for them. So to plan ahead, one can refer to the forecasts that are issued ahead of time like, what is the chance of rain tomorrow? We assume a forecaster who converts historical and current information into forecasts, and we’re concerned with if someone is a good forecaster?…

Continue reading →

No sure loss, calibration, and insurance

The field of machine learning is rife with uncertainty quantification. In fact, the standard approach to prediction is based on quantifying the confidence over the space of possible outcomes, e.g. the confidence over the whole space of vocabulary for the next token prediction in text generation. Despite the wider scope of quantifying uncertainty in machine learning pipelines, the consensus over what defines a good notion of uncertainty remains an unsettled topic. Calibration, in my limited viewpoint, takes the cake of a super simple, highly minimal, and yet very powerful notion of uncertainty quantification. And maybe this is the reason that it has remained an object of study among machine learning practitioners for years. Despite the promise and the popularity of calibration, I’ve always felt a tinge of discomfort every time I’ve heard or read about it. In this note, I’ll try to articulate why it could have been like that, and why this should not be the case…

Continue reading →