Thought Prompts

Want to know if A beats B? First ask if doing A changes how you do B (or vice versa)? If the answer is “Yes”, use different people for each. Give one group A, the other B, and measure the same outcome once. That’s an independent two-sample comparison. No memory effects. No practice carryover. Just […]

Computers are great at arithmetic. People are great at judgment. Give the calculator the heavy lifting, keep the meaning for yourself. Whether you use SPSS, Excel, or a notebook, the workflow is the same: define the question, choose the right test, run it, then decide. The software can’t pick your hypothesis, your confidence level, or

Big p = “Not Yet” A large p-value doesn’t crown the null; it says today’s evidence can’t clear your risk line (alpha). Treat it as “not yet.” Tighten measures, strengthen design, or raise n. Then rerun. Small p, Small Claim A tiny p-value rejects “no effect.” It doesn’t prove a big effect, a useful effect,

One result is a flare, not a sunrise. Surprising claims (delightful or uncomfortable) earn replication. Make round two stricter: preregister, tighten measures, reduce researcher degrees of freedom, vary the setting or sample. If it’s real, it travels. If it’s fragile, you just saved months of theater. Build replication into the roadmap like QA: default, not

“Statistically significant” and “worth doing” are different currencies. A huge sample can make dust look like diamonds. After p beats alpha, ask: How big is the effect and does it justify action? Will a 0.4% lift cover engineering time, UX complexity, support load, and cognitive tax? Flip it, too: a chunky, user-loving effect with borderline

Don’t let noise pick your narrative. If theory says “up,” preregister a one-sided test (greater than 0) before collecting data. If you genuinely don’t know, choose two-sided (not equal to 0). One-sided buys power only when you commit; post-hoc direction isn’t power, it’s wishful thinking. In all cases, the null is still “effect = 0.”

Alpha = .05 is not sacred. It’s a policy choice. What’s the cost of a false alarm here… money, time, safety, trust? Choose alpha to match reality: .10 for quick screens, .05 for routine work, .01 (or .001) when stakes are steep. Declare it before the run. Then compare: p below alpha means you’ll proceed;

A p-value isn’t a victory lap. It’s a yardstick against a world where your treatment does nothing. You write down that world (the null), set your alpha (your risk budget for false alarms), and then ask: “If nothing were true, how surprising is what we saw?” If p < alpha, you’ve earned evidence to move

1) Two Levers, One Bill  You can squeeze the margin by lowering confidence or raising n. One costs honesty. The other costs effort. Pick your bill and pay it on purpose. 2) Use t. Full Stop.  We don’t know σ. We almost never do. That’s why t exists. Feed it df = n − 1, take the

The recipe stays simple: statistic ± (critical value × standard error). Tools can spit out the numbers. Your job is the story. Name the parameter. State the level. Share both versions… “lower–upper” and “estimate ± margin.” Then attach a verb: staff, budget, delay, pull forward. Machines calculate. You commit.