Grafen & Hails: Modern Statistics for the Life Sciences
17.06.03 Robert D. Arbeit, Clinical Research, Paratek Pharmaceuticals, Boston, MA, USA
Chapter 15, page 297
Q: I believe the results shown in Box 15.9 are incorrect as regards the results for the variable SEX. As shown, SEX alone is NOT significant, which contradicts the analysis in Chapter 6 (p99-102).
A:Robert is right to notice a difference between the conclusions about SEX that you would be tempted to draw from the output in Box 6.2 on page 100, on the one hand, and from Box 15.9 on page 297, on the other. However, there is no mistake in the output, and the explanation is that the test for SEX in Box 15.9 is an invalid test and should not be made.
This is to do with marginality, as explained in Chapter 10 on pages 192 and 193. Specifically, the Adj SS for SEX in Box 15.9 has been adjusted for a "higher order term", namely SEX*WEIGHT, and so the test for SEX is invalid.
Robert might wonder why we show tests that are wrong, and the answers are the same as given in a previous response. (1) At the stage in the book the exercise is intended for, we have introduced interactions but haven't yet got to marginality, so we can't explain at that point what is going on (2) In some packages, including Minitab and SPSS, the Adj SS are tested by default and, so long as you are interested only in the interaction, it is therefore easiest to do that analysis. (3) Packages do produce such p-values, and it is useful for users to learn to notice them and ignore them.
This is the second query about these invalid tests, so they are obviously a sticking point for intelligent readers. We will consider putting in a mysterious warning such as "Readers may notice that the main effect of SEX has become non-significant, but we must defer an explanation of this phenomenon until we introduce 'marginality' in Chapter 10."