Careers
Part of Offline campaign measurement: a practical process from decision to documented result
3 things to understand about offline campaign measurement mistakes
Three offline campaign measurement mistakes to understand first: undefined exposure and denominators, missing comparisons, and rules changed after the results.
What to take away
- A number without an event definition and denominator is not a usable rate.
- Post-result rule changes invite selective conclusions.
- More tracking does not repair a missing comparison or an immature outcome.
Offline campaign measurement mistakes usually begin before analysis. The three below cause the most damage, and the full list of defects follows them.
1. Exposure and response are not defined
Mailers, flyers and posters are often reported as reach. A printed piece that was delivered may never have been seen. A scan is a technical event, not a lead. Both mistakes inflate the numerator or the denominator of a rate, and the rate looks precise while being meaningless.
Fix it with one definition per event. Label delivery as delivery. Count a response only when it is a qualified action, such as a booked call or a redeemed offer. Use unique promo codes, call-tracking numbers, or QR codes that route to a tracked page, so each response has a source.
2. There is no comparison group
A campaign run in every market at once cannot show what would have happened without it. Sales can rise for reasons unrelated to the mailer, such as a holiday, a price change or a competitor's exit. The trend then becomes a campaign effect in the report.
Use a geographic holdout, where comparable areas receive no mail, or a matchback analysis that compares responders against the customer file. Choose the comparison before launch, check for exposure through other routes, and narrow the claim to what the design supports.
3. Rules change after the results arrive
Teams often pick a measurement window, an attribution rule or an exclusion after they have seen which version looks better. That is selective reporting, even when no one intends it. The same applies to cost: a report that leaves out print, postage or distribution costs overstates the return.
Freeze the window, the attribution rule and the cost scope before the campaign starts. Record any later change with its date and reason, so a reviewer can see what moved and why.
The full list of defects
| Mistake | Consequence | Correction |
|---|---|---|
| Pieces equal reach | Exposure is overstated | Label delivery and estimate exposure separately |
| Scans equal leads | Technical events become demand | Require a qualified customer action |
| No identity rule | People and events are double counted | Publish reconciliation order |
| No comparison | Trend becomes campaign effect | Use a credible counterfactual or narrow claim |
| Window chosen later | Favorable timing is selected | Freeze windows before analysis |
| Only matched records | Missingness disappears from report | Publish match rate and loss reasons |
| Revenue equals value | Returns, margin, and cannibalization vanish | Use mature incremental value |
| One precise rate | Uncertainty and sensitivity are hidden | Report range, assumptions, and thresholds |
Check the data before the result
- Search for staff tests, bots, retries, and repeated customers
- Compare source totals before and after every join
- Inspect missing records by audience, place, and outcome
- Record price, inventory, staffing, weather, and other media changes
- Recompute under reasonable windows and attribution rules
- Correct claims that exceed the design
A failed study can still produce valuable operational evidence. State that delivery broke, identity was incomplete, spillover was uncontrolled, or the outcome had not matured. Do not rescue the campaign by changing the question to whichever metric moved.
Prove the correction
A mistake is not closed when a dashboard changes. Trace the correction from source record through processing, reporting, exports, and the decision owner.
Preserve the prior value, reason, date, reviewer, and affected actions. Then test whether the same defect can recur. This turns an error list into an operating control and shows later readers the published conclusion was repaired.
Keep the evidence record beside the decision so a reviewer can reproduce the reasoning without memory. Record rejected options and the chosen path, because the original constraint may later change. Set the next review date and name the change that would trigger an earlier check. For the strategy side of these choices, see offline marketing strategy mistakes.
Common questions
Is last-touch attribution a mistake?
It is a usable convention when labeled, but a mistake when presented as causal proof.
Can a very large dataset remove bias?
No. More records can make an estimate more precise while selection, measurement, identity, or confounding bias remains.
What if no credible comparison exists?
Report implementation and observed outcomes with limits, avoid lift claims, and design a better next test.
