If one test risks a 5% false positive, what happens when you run ten? You don’t get ten times the truth… you get nearly a coin-flip chance of fooling yourself. Multiple testing quietly inflates Type I error… turning noise into “discoveries.” The fix isn’t magical; it’s methodological. Either adjust your alpha (Bonferroni, Holm, FDR) or use a design that asks the one big question once. Enter ANOVA: a single model that compares several means while guarding the overall error rate. The real discipline, though, is upstream… write a single, decisive question before you touch the data. Lock your outcomes. Define what “difference” means (effect size and direction) ahead of time. Then, when curiosity whispers “just one more test,” you can answer with a plan, not a p-value. More tests feel productive. Fewer, pre-specified tests produce decisions you can defend.
