Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

The test statistics underpinning several methods for combining <i>p</i>-values are special cases of generalized mean <i>p</i>-value (GMP), including the minimum (Bonferroni procedure), harmonic mean and geometric mean. A key assumption influencing the practical performance of such methods concerns the dependence between <i>p</i>-values. Approaches that do not require specific knowledge of the dependence structure are practically convenient. Vovk and Wang derived significance thresholds for GMPs under the worst-case scenario of arbitrary dependence using results from Robust Risk Analysis (RRA). Here I calculate significance thresholds and closed testing procedures using Generalized Central Limit Theorem (GCLT). GCLT formally assumes independence, but enjoys a degree of robustness to dependence. The GCLT thresholds are less stringent than RRA thresholds, with the disparity increasing as the exponent of the GMP ( <i>r</i>) increases. I motivate a model of <i>p</i>-value dependence based on a Wishart-Multivariate-Gamma distribution for the underlying log-likelihood ratios. In simulations under this model, the RRA thresholds produced tests that were usually less powerful than Bonferroni, while the GCLT thresholds produced tests more powerful than Bonferroni, for all <i>r</i>> - ∞. Above <i>r</i>> - 1, the GCLT thresholds suffered pronounced false positive rates. Above <i>r</i>> - 1/2, standard central limit theorem applied and the GCLT thresholds no longer possessed any useful robustness to dependence. I consider the implications of these results in the context of various interpretations of GMPs, and conclude that the GCLT-based harmonic mean <i>p</i>-value procedure and Simes' (1986) test represent good compromises in power-robustness trade-off for combining dependent tests.

Original publication




Journal article


Wellcome open research

Publication Date





Big Data Institute, Nuffield Department of Population Health, University of Oxford, Oxford, OX3 7LF, UK.