“Statistically significant” and “worth doing” are different currencies. A huge sample can make dust look like diamonds. After p beats alpha, ask: How big is the effect and does it justify action? Will a 0.4% lift cover engineering time, UX complexity, support load, and cognitive tax? Flip it, too: a chunky, user-loving effect with borderline p might earn replication, not burial. Statistics answers “Is there something?” Strategy answers “Should we do something?” Mature teams run both tests, in that order. Celebrate effects big enough to matter. Retire trivia, even when it glitters.
