Statistical Modeling, Causal Inference, and Social Science: How should unproven findings be publicized?

How should unproven findings be publicized?

A year or so ago I heard about a couple of papers by Satoshi Kanazawa on "Engineers have more sons, nurses have more daughters" and "Beautiful parents have more daughters." The titles surprised me, because in my acquaintance with such data, I'd seen very little evidence of sex ratios at birth varying much at all, certainly not by 26% as was claimed in one of these papers. I looked into it and indeed it turned out that the findings could be explained as statistical artifacts--the key errors were, in one of the studies, controlling for intermediate outcomes and, in the other study, reporting only one of multiple potential hypothesis tests. At the time, I felt that a key weakness of the research was that it did not include collaboration with statisticians, experimental psychologists, or others who are aware of these issues.

Peter Hoff

In addition to Kanazawa I would also blame the
review process of scientific journals. I don't
know the review procedures of JOTB, but I imagine
it isn't too different from those of journals
that I'm familiar with. Articles are not
refereed by a large group of highly qualified
and interested individuals, they are refereed by the first two or three people who the AE happens
to talk into providing a review.
These referees may lack the time, interest and/or
ability to provide a thorough review of the
scientific claims of the article.
Additionally, if the AE recruits reviewers
from the author's bibliography, it is
likely that the pool of reviewers will
not be sufficiently skeptical of the author's
claims.

Is there an alternative? I'm hoping that
dynamical peer review (e.g. www.naboj.com)
will evolve to the point that it can replace
our current system.

Peter Hoff

In case you haven't seen this:

http://poynder.blogspot.com/2006/10/open-access-death-knell-for-peer.html

John Fleck

This is most troubling for a journalist (me). Given the intro, I assumed the punch line would be some hazy gray literature thing that had not been peer reviewed.

conchis

Personally, I think you're being a little too generous to Kanazawa. Particularly now that you've drawn his attention to problems with his statistical analysis, it seems slightly disingenuous of him to continue making the forceful claims he seems to be making. This is even more so if he knows that he's writing for an audience that may not be able to assess those claims adequately themselves - in such a case I think there's even more of an onus on him to state his claims carefully...

shawn

I agree with the previous comment that you may be too generous - especially given that Kanazawa is a member of the methodology institute at LSE and teaches graduate statistics courses in the social sciences.

John

Andrew, that is one well-written letter. Thank you for linking to it. ^_^

The "Psychology Today" article is disturbing for more than just beauty/child-sex issue: Just prior to the Iraq war, "Frontline" reported that the most prolific group of suicide bombers were the Tamil Tigers, fighting in Sri Lanka. Tamils, the people who invented the suicide-bomber explosive-vest, are mostly Hindu; approximately 5% are Christian or Muslim (see Wikipedia under "Tamil").

I don't know if the Tamil Tigers have lost the top spot in the last few years; however, Wikipedia reports an estimated 6.5 million Sri Lankan Tamils. Compared to the number of Muslims from which to draw suicide bombers, it certainly seems unlikely that something about Islam can be blamed for suicide bombers.

Indeed, the article suggests that suicide bombing is a strategy for success in the mating game. One suspects that logic only really works for swamp dragons on Discworld.

Carl

There's also a good discussion of the Pych. Today article and some background on Kanazawa's other controversial work here:
http://echidneofthesnakes.blogspot.com/2007_07_01_archive.html#8291138433810237995 .

simon

Andrew-

I applaud your willingness to raise the question. I also agree with a number of the comments that question Kanazawa's integrity on the matter. This type of work results in science and statistics to be dismissed by the general public. All the downside so our friend Kanazawa can become a celebrity. I'm not buying

Igor Carron

Andrew,

Self doubt is a good thing when it comes to addressing the methods as opposed to the actual phenomena at hand. However, as you know in the social science world, there is an enormous societal concern that trumps letting bad results get press/book coverage.

Igor.

georgia

I would like to see reputable journals that focus on critiquing previous work. This would provide a nice forum for people to respond to studies like Kanazawa's and would probably encourage researchers like Kanazawa to use more solid methods. And, people that are interested in or skeptical of articles like Kanazawa's would know where to look for critiques.
(Similarly, I would like to see a journal in my field that publishes results that confirm null (and less interesting) hypotheses.)

Keith O'Rourke

I believe the community has to empirically show that the claims do not replicate

We recently got lucking in genetics because many of the false positive published claims could quickly be checked by other labs that already had the data in hand ... and the false positive claims were made by well respected senior (basic) scientists in journals like Nature (now some of these journals are thinking of delaying publication until claims have been replicated by other groups.)

As for "reporting only one of multiple potential hypothesis tests." I am aware of a survey of clinical researchers that suggest 50% just don't believe this is problem. Empirical studies that track studies from ethics approval forward are trying to correct this misunderstanding (Altman and Chan).

"At the time, I felt that a key weakness of the research was that it did not include collaboration with statisticians"

My prior would be centered at 35% with little mass below 10%. What perecentage of statisticians believe that "reporting only one of multiple potential hypothesis tests." is a problem?

Unfortunately it is hard to do good empirical studies on questions like this and often they are not needed for those who understand that "2 + 2 = 4" (in an ideal world we would not need them) but they are needed for the wider community that must "step in" on these issues.

"getting off my soap box now"

Keith

Dan tdaxp

Really?