BLOG

Big p = “Not Yet” A large p-value doesn’t crown the null; it says today’s evidence can’t clear your risk line (alpha). Treat it as “not yet.” Tighten measures, strengthen design, or raise n. Then rerun. Small p, Small Claim A tiny p-value rejects “no effect.” It doesn’t prove a big effect, a useful effect, […]

One result is a flare, not a sunrise. Surprising claims (delightful or uncomfortable) earn replication. Make round two stricter: preregister, tighten measures, reduce researcher degrees of freedom, vary the setting or sample. If it’s real, it travels. If it’s fragile, you just saved months of theater. Build replication into the roadmap like QA: default, not

“Statistically significant” and “worth doing” are different currencies. A huge sample can make dust look like diamonds. After p beats alpha, ask: How big is the effect and does it justify action? Will a 0.4% lift cover engineering time, UX complexity, support load, and cognitive tax? Flip it, too: a chunky, user-loving effect with borderline

Don’t let noise pick your narrative. If theory says “up,” preregister a one-sided test (greater than 0) before collecting data. If you genuinely don’t know, choose two-sided (not equal to 0). One-sided buys power only when you commit; post-hoc direction isn’t power, it’s wishful thinking. In all cases, the null is still “effect = 0.”

Alpha = .05 is not sacred. It’s a policy choice. What’s the cost of a false alarm here… money, time, safety, trust? Choose alpha to match reality: .10 for quick screens, .05 for routine work, .01 (or .001) when stakes are steep. Declare it before the run. Then compare: p below alpha means you’ll proceed;

A p-value isn’t a victory lap. It’s a yardstick against a world where your treatment does nothing. You write down that world (the null), set your alpha (your risk budget for false alarms), and then ask: “If nothing were true, how surprising is what we saw?” If p < alpha, you’ve earned evidence to move

1) Two Levers, One Bill  You can squeeze the margin by lowering confidence or raising n. One costs honesty. The other costs effort. Pick your bill and pay it on purpose. 2) Use t. Full Stop.  We don’t know σ. We almost never do. That’s why t exists. Feed it df = n − 1, take the

The recipe stays simple: statistic ± (critical value × standard error). Tools can spit out the numbers. Your job is the story. Name the parameter. State the level. Share both versions… “lower–upper” and “estimate ± margin.” Then attach a verb: staff, budget, delay, pull forward. Machines calculate. You commit.

Two gates before any math: random sample and normal-or-large. No random sample? Pass. Tiny sample with no reason to believe “bell curve”? Pass. When the gates open, use t, not wishful thinking… because we almost never know the population spread. Degrees of freedom = n − 1. (Just like the last diner at the buffet:

Ninety percent confidence is not a guarantee. It’s a seatbelt. Nine trips out of ten, you arrive with your conclusion still buckled to the truth. One trip, something weird happens… snow day, breach, surprise. The miss isn’t failure; it’s the known cost of not pretending you’re psychic. Plan for the nine. Prepare for the one.