Senn-isms

From Chance News come these quotations from Stephen Senn’s book Statistical Issues in Drug Development:

“Most Trials are unethical because they are too large.” Page 178.

“Small Trial are unethical.” Page 179.

“A significant result is more meaningful if obtained from a large trial.” Page 179.

“A given significant P-value is more indicative of the efficacy of a treatment if obtained from a small trial.” Page 180.

“For a given P-value, the evidence against the null hypothesis is the same whatever the size of the trial.” Page 182.

Perhaps Stephen can explain? He’s a funny guy, so I don’t really know if these are jokes or if there are more serious points he’s trying to make by these contradictions.

10 thoughts on “Senn-isms

  1. 1) is true, because "too large" means "too many people exposed to an experimental treatment that might be worse than known treatments in some way." Almost a definition, though. We exposed an extra 100 people, knowing that it improved our chances of learning something by very little.

    2) is true, because "too small" means people were exposed… but there wasn't any real chance of learning anything from the study.

    3), 4), and 5) sure look like contradictions to me, too.

  2. 3 and 4 are obviously not contradictory. Significant effects in large trials require large numbers of failures to overcome. But, holding p value constant, effects must be larger in small trials than in large trials… it takes huge effects in a small trial to get the same p value as some arge trial. But if you're going to use p values as evidence, which you shouldn't (Royall 1997) the evidence they give is the same.

  3. The 180 / 182 comments make sense: in terms of statistical significance, you only need to look at the p-value. But in terms of the medical significance, a smaller sample with the same p-value must have a much larger point estimate on the size of the effect.

  4. So, to summarize the above comments:

    Statement 1 is about "most trials" whereas statement 2 is about "small trials", so there's no inherent contradiction. Statement 3 is about what p-values say about effect sizes whereas statement 4 is about what p-values say about hypothesis testing, and again there is no inherent contradiction. (I'm down with Royall on p-values — I just think he shouldn't have stopped at likelihood, but gone all the way to Bayes.)

  5. Regarding 3&4, let's remember that p-values, under the null hypothesis, are uniformly distributed.

    So maybe they tell us nothing about anything.

  6. Alex F – you ruined our chances for an all "John/Jonathan" comment list!

    The smaller sample with the same p-value must have a much larger point estimate on the size of the effect, true, but a much less accurate one as well. Am I overlooking something fundamental here?

  7. If we condition on a p-value of 0.005, with a small sample we may have a 99% confidence interval on a treatment effect of [0, 10], say, but with a much larger sample the confidence interval may be [0, 1]. The latter would seem to be much more indicative of the efficacy of the treatment than the former, contradicting point #4. Less indicative of a potentially very efficacious treatment; more indicative of what the efficacy actually is.

  8. It's flattering to find so much interest in the book. However,…in the preface to the first edition it says
    "I became convinced that statisticians pay their non-statistical colleagues a disservice if they try to gloss over genuine disagreements."

    and in the preface to the second edition

    "It is appropriate for me to repeat a general warning about the book that Carl-Fredrik Burman has drawn to my attention. A number of the section heading contain statements of position. For example, Chapter 7 has a section, ‘The propensity score is a superior alternative to adjusting for confounders than analysis of covariance’. You will get a very misleading impression of the message of the book if you take these as being my position. This is a book about issues and where such statements are made it is nearly always because, as is the case with this one, I wish to take issue with them."

    In other words, the headings are statements that you will find given and justified in the statistical literature. There is a sense in which they are true – an exact moderately significant P-value is more impressive if it comes from a small trial, for example, whereas the result 'significant at the 5% level' is more impressive if it comes from a large trial. However, they do not represent, when taken out of context, my view.

    Stephen

Comments are closed.