User:Talgalili/sandbox/Closed testing procedure

In statistics, the closed testing procedure is a general method for controlling the experimentwise or multiple type I error rate when multiple hypothesis have to be tested ,i.e. when performing more than one hypothesis test simultaneously.

The intrinsic feature of the closed testing procedure is to refer a set of hypotheses which are closed under intersection, and that each test is of level α. The procedure controls the familywise error rate for all k hypotheses at level α in the strong sense.

Theoretical background - the problem of multiple testing

The contest with testing simultaneously several hypotheses differs from the concept of dealing with only one hypothesis. In single hypothesis, under the assumption that the null hypothesis is true, the probability of rejecting it is less or equal level α. There is a need for a similar concept also for handling multiple hypotheses, because if we will apply individual tests at a given level α, then, under the assumption that all null hypothesis are simultaneously true - the probability of falsely rejecting one of them would exceed the given level α. For example, If all tests are independent, the probability would be $1-(1-\alpha )^{k}$ where ${k}$ is the number of hypotheses.at some point, this probability would be equal to 1.

One of the first general methods of alpha adjustment for multiple comparisons early has been the Bonferroni correction. That means that for testing k hypotheses, the single tests should be performed at the level α/k.

The closed testing principle

Let $X$ be a random variable with distribution $P_{\theta }\in \Omega$. Let $H={\{h_{p}}\}$ be a set of null hypotheses closed under intersection: ${h_{i}},{h_{j}}\in \mathbb {H}$ implies ${h_{i}}\cap {h_{j}}\in \mathbb {H}$. For each ${h_{p}}$ let ${\phi _{p}}(X)$ be a level α test (i.e.the overall type I error rate is α). Any null hypothesis ${h_{p}}$ is tested by means of ${\phi _{p}}(X)$ if and only if all hypotheses ${h}$ that are included in ${h_{p}}$ and belonging to ${H}$ have been tested and rejected. In other words, every single hypothesis of this set is tested using valid local level α tests. Hypothesis is only rejected if its test is significant and the tests of all single hypotheses which are subsets of it is significant.

Proposition: The closed testing principle controls the familywise error rate for all the ${p}$ hypotheses at level α in the strong sense.That is, The probability of making no type I error is at least 1-α.

Proof: If we denote by ${A}$ the event that at least one true ${h_{j}}$ is rejected, and by ${B}$ the event that one true hypothesis is rejected. In order for the closure procedure to reject (the correct) ${h_{j}}$, all hypothesis implying it should be rejected, and in particular the hypothesis in the event ${B}$. That means, $A=A\cap B$.

$FWER=\Pr _{\theta }(A)=\Pr _{\theta }(A\cap B)=P(B)P(A|B)\leq \alpha$

The procedure

${1.}$ Test every member of the closed family by a α-level test.

${2.}$ Reject a basic hypothesis if its corresponding α-level test rejects it, and every intersection hypothesis that includes it is also rejected by its α-level (see examples).

Examples

Suppose there are three hypotheses H₁,H₂, and H₃ are to be tested and the overall type I error rate is 0.05. Then H₁ can be rejected at level α if H₁ ∩ H₂ ∩ H₃, H₁ ∩ H₂, H₁ ∩ H₃ and H₁ can all be rejected using valid tests with level 0.05.

Now, suppose there are four hypotheses H₁,H₂,H₃ and H₄ and we use pairwise problem. The resulting closed set of hypotheses includes those single hypotheses:

H₁ ∩ H₂ ∩ H₃ ∩ H₄,

H₁ ∩ H₂ ∩ H₃, H₁ ∩ H₂ ∩ H₄, H₁ ∩ H₃ ∩ H₄, H₂ ∩ H₃ ∩ H₄, H_1,2 ∩ H_3,4, H_1,4 ∩ H_2,3, H_1,3 ∩ H_2,4,

H₁ ∩ H₂, H₁ ∩ H₃, H₁ ∩ H₄, H₂ ∩ H₃, H₂ ∩ H₄, H₃ ∩ H₄.

In the closed test procedure, everyone of those single hypothesis is tested at level α. The hypothesis is rejected only if its test is significant and the tests of all single hypothesis which are subsets of it. That is the reason why the closed test procedure keeps the multiple level α.

For example, the hypothesis H₂ ∩ H₄ is only rejected if all the tests for the hypothesis H₁ ∩ H₂ ∩ H₃ ∩ H₄, H₁ ∩ H₂ ∩ H₄, H₂ ∩ H₃ ∩ H₄, H_1,3 ∩ H_2,4 and H₂ ∩ H₄ are significant.

In the sketch from the right side it is possible to see the hierarchy of the procedure. The single hypothesis are tested in the order of the hierarchy, starting with the hypothesis on the highest level (H₁ ∩ H₂ ∩ H₃ ∩ H₄ in our case). If this single hypotheses is significant, the hypothesis below will be tested (marked with a red arrow). Otherwise it will be not be tested and regarded as non significant. A single hypothesis will be only tested if all its super-sets one level above are significant.

Special cases

The Holm–Bonferroni method is a special case of a closed test procedure for which each intersection null hypothesis is tested using the simple Bonferroni test. As such, it controls the familywise error rate for all the k hypotheses at level α in the strong sense. Additional special case is Hochberg's step-up procedure(1988) which has the same constant as for Holm–Bonferroni method but the direction is opposite, affecting the stopping location, and as a result Hochberg's method more powerful than Holm–Bonferroni. However, while Hochberg’s is based on the Simes test (1987) and thus holds only under independence (and also under some forms of positive dependence), Holm’s is based on Bonferroni with no restriction on the joint distribution of the test statistics.

Multiple test procedures developed using the graphical approach for constructing and illustrating multiple test procedures are a subclass of closed testing procedures.

See also

• Multiple comparisons
• Holm–Bonferroni method
• Bonferroni correction
• Hochberg's step-up procedure(1988)

References

Category:Hypothesis testing Category:Statistical tests Category:Multiple comparisons