Instrument before you launch

The feature is live and somebody asks whether it worked. You add the events that afternoon, wait two weeks, and come back with numbers.

The usual version of this chapter says the counterfactual is gone at that point, and that is too strong to print. You can still create a control group after launch by putting the feature behind a flag and turning it on for half your users. The comparison survives.

What does not survive is the before, and the price of losing it turns out to be measurable.

What retrofitting actually destroys

Two things, and one of them has a number attached.

Your history is not backfilled. Google’s documentation puts it in one sentence: “Modified and new events do not apply to historical data.” The events you define today start existing today. Whatever the world looked like last month is not recoverable, because nothing was writing it down.

And pre-period data is worth roughly half your sample. This is the part worth knowing. Microsoft published a technique called CUPED that uses data from before an experiment to reduce variance in the result. Their finding: “Three important recent experiments showed variance reductions of 45%, 52% and 49% with one week of experiment and one week of pre-experiment data.”

Roughly half the variance removed, by having measured the same users beforehand. In practical terms that is the same statistical power on half the users, or in half the time. So instrumenting late does not merely delay you. It roughly doubles the traffic or the duration you need to reach the same confidence, forever, on that metric.

That is the actual bill, and it is why this is a precondition rather than a task. As Kohavi, Tang and Xu put it: “Before you can run any experiments, you must have instrumentation in place to log what is happening to the users and the system.”

It drifts unless it is a rule

This is not a discipline problem you can solve by intending to do better. GitLab published their own audit of it, and the numbers are worth sitting with: they were shipping around 30 features per milestone, and analysis found “only 50% of our premium features are currently instrumented.”

A company that publishes its handbook, with an analytics team, found half its paid features had no measurement on them. Whatever your team’s number is, it is not better than that by accident.

The move

Write the events in the same pull request as the feature, and treat a feature with no events the way you would treat one with no tests.

BEFORE                          AFTER

PR #4412  "Add bulk edit"       PR #4412  "Add bulk edit"
  src/routes/admin.tsx            src/routes/admin.tsx
  src/lib/bulk.ts                 src/lib/bulk.ts
  tests/bulk.test.ts              tests/bulk.test.ts
                                  analytics/events.ts   <- same PR

  ship. then, six weeks           bulk_edit_opened
  later, someone asks             bulk_edit_submitted
                                  bulk_edit_failed
  add events. wait two
  weeks. no baseline, and         object_verb_pasttense, lower snake.
  double the traffic needed        one schema, one place, reviewed
  for the same confidence          like any other code

The naming matters more than it looks. Pick one convention, object_action in the past tense, lower snake case, and hold it. Analytics event names are the one part of your codebase nobody refactors, because renaming an event breaks every historical query that referenced it. You are choosing these names once.

What it costs

Instrumentation is real work in every pull request, and it is the work people cut first when a sprint is tight. There is no version of this that is free. The honest framing is the same as move fifteen’s: it is a scope decision, so make it explicitly rather than by omission.

And the payoff is not uniform. CUPED’s own paper names its exception: “revenue-per-user, where CUPED reduced the variance by less than 5% due to the low correlation of revenue-per-user between the pre-experiment and the experiment periods.” Pre-period data helps most for metrics where a user’s past behaviour predicts their future behaviour, and barely at all for lumpy ones. Do not promise your team a halving on every metric.

Also, the data you do collect expires. Google’s retention for user-level data is 2 or 14 months, and for other event data up to 38 on the paid tier. A baseline you captured three years ago may not be there. Check your retention setting before you rely on history you assume exists.

Try this week

Open the last three features your team shipped and check whether each one emits a single event.

Most teams find one of the three does. That is the finding, and it is more persuasive to a sceptical lead than any argument about rigour, because it is about their own product.

Then take whatever you are building next and add one line to the pull request template:

Events emitted: …

Not a schema, not a taxonomy, not a project. One line that is empty until somebody fills it, in the same review where you would notice a missing test.