Disagree with a number, not an opinion

You think the migration pays for itself inside two quarters. Your product lead thinks it is six months of work with nothing to show a customer. You have each said this twice, more carefully the second time, and nothing moved.

The instinct at this point is to argue better. Sharper framing, a diagram, an ally in the room. It does not work, and it does not work for a reason worth understanding: you are not disagreeing about a fact you both have. You are comparing two predictions, and predictions about software are much worse than the people making them believe.

Confident predictions about software are usually wrong

Microsoft’s experimentation teams have put numbers on this. Across their experiments, only something like one idea in five to one in ten produced the improvement it was designed to produce, with an average prior around one in three.

Two of their cases are the whole argument. An advertising headline change that the program managers ranked as low priority, and left sitting for months, turned out to be worth about 12% of revenue. A longer, richer results page that everyone expected to win measured 1.8% down on revenue and was not released.

Nobody in either story was stupid. They were confident, experienced, and reasoning from a model of users that happened not to match the users.

And before this reads as a chapter about other people being overconfident: METR put sixteen experienced open-source developers on 246 real tasks from their own repositories. The developers predicted AI tooling would make them about 20% faster. They were 19% slower, and still believed they had been faster afterwards. The experts asked to forecast the result got the sign wrong too.

We are not good at this. That is not a character flaw, it is the normal condition, and it is why the meeting keeps ending the way it does.

The move

Turn the disagreement into a quantity somebody could measure this week, then measure it.

Note the goal. Not to win the argument. To stop having it.

Getting a number cheaply

The objection is that measuring properly takes longer than the work. Usually it does not, because you do not need a proper measurement, you need one good enough to separate two positions that are miles apart.

The canonical demonstration is Fermi at the Trinity test. His own account: “I tried to estimate its strength by dropping from about six feet small pieces of paper before, during and after the passage of the blast wave… The shift was about 2½ meters, which, at the time, I estimated to correspond to the blast that would be produced by ten thousand tons of T.N.T.” Scraps of paper, in the desert, same order of magnitude as the instrumented answer.

Douglas Hubbard’s Rule of Five is the version you can use on a Tuesday: “There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population.” Five. Not a study, not a quarter of telemetry. Time five deploys. Ask five support agents. Read five tickets.

Decomposing an estimate into parts does help, though be honest about how much: in a controlled study the improvement from Fermi-style decomposition was real but modest, and its confidence interval crossed zero.

THE DISAGREEMENT              THE QUANTITY, AND HOW TO GET IT

"the migration pays for       how many engineer-hours per week go to
 itself in two quarters"      working around the current schema?

"it's six months with           -> read five weeks of merged PRs,
 nothing to show"                  count the ones that touch the shim.
                                   two hours, today.

"support is drowning in       what fraction of tickets last month
 password resets"             mention a reset?

                                -> sample five days, count. one hour.

  each side names the number it expects BEFORE the measurement,
  in writing. that is the part that makes it decidable.

That last line matters more than the method. Write down what each of you expects before you look. Otherwise the result gets absorbed into whatever you already believed, and you have spent the afternoon buying nothing.

Numbers are not actually very persuasive

I have to be straight about the weakest part of this move, because the usual version of this advice oversells it.

There is little good evidence that presenting a quantified case changes a stakeholder’s mind. The best field experiment I could find is not from software: a randomised trial across 3,218 nail salons gave some of them a personalised report showing their price against their nine nearest competitors. Treatment firms were three percentage points more likely to change their prices. Real, causal, and small.

The tempting counter-claim, that numbers actively backfire on people whose identity is invested in the answer, is also not safe to lean on: the best-known result in that literature failed a preregistered replication.

So do not expect the number to convert anyone. Expect it to convert the question, from whose judgment we trust into what actually happened, and those are very different meetings.

What it costs

You have to accept the answer when it goes against you. This is the entire price, and it is higher than it sounds. If you propose the measurement, you are bound by it. Do it once, honour a result you hated, and your next proposal is treated completely differently. Wriggle out of it once and this move is permanently unavailable to you.

Some things genuinely cannot be measured this week, including most architectural decisions whose payoff is three years of not being stuck. Do not pretend otherwise, and do not force a bad proxy on a real question. A measurable proxy that misses the point is worse than an honest disagreement, because it settles the argument in the wrong direction with authority.

Try this week

Take the disagreement you have had twice, and write one sentence: we disagree about X, and if X were above N I’d be wrong.

If you cannot fill in N, that is the finding, and it is usually the reason the argument is stuck. Neither of you has been claiming anything falsifiable.

Then get a bad, cheap version of the number today. Five samples. An hour. Send it to the person you disagree with, with both predictions written down first, and let it decide.

You will be wrong some of the time. That is not the failure case. That is the whole reason the method is worth anything.