Flexible Multiple Testing with the FACT Algorithm

Modern high-throughput science often leads to multiple testing problems: researchers test many hypotheses, wishing to find the significant discoveries. The development of flexible multiple testing methods is thus a central problem in statistics. In this paper, we introduce the new Fast Closed Testing (FACT) method for multiple testing, controlling the family-wise error rate. Our method relies on symmetry and monotonicity to enable the classical closed testing principle in the important setting of large datasets. As the closed testing principle is more than 40 years old, we find it surprising that this simple and fundamental algorithm has not been described before. Our FACT method is general and flexible, and can be used to design powerful new architectures for multiple testing. We showcase it by proposing the Simes-Higher Criticism fusion test, which is powerful for detecting both a few strong signals, and also many moderate signals. We illustrate the method in simulations and in a genome-wide association study of coronary artery disease, and obtain more power than with existing methods.