Stephen Dubner writes about a professor of economics who makes zillions of dollars consulting. I’m not surprised by this, because my impression is that legal consulting is an extremely inefficient market. In several of the cases I’ve consulted on, the statistical expert (typically not actually a statistician) has been truly incompetent, in one case not being certified as an expert in the relevant area by the court, and in another case–which was truly memorable–dividing by N (the population size) rather than n (the sample size) in computing the estimation variance from a random sample. (I was really looking forward to the exchange with this guy in the courtroom–I mean, what could he say, he’s a sampling expert and divided by the wrong N?–but, like most cases, it got settled before reaching court.)
I can give other stories, but the key point is that lawyers hire incompetent statistical experts, even in cases that are important. It’s gotta be worth their money to hire better consultants, but presumably they can’t find them. Actually, I think that I’ve probably cost less than the opposing consultant in every case I’ve worked on, since, despite my high hourly rate, I’m trying to minimize my consulting hours, whereas I imagine that professional consultants are, if not trying to maximize hours, at least to keep their business going. But most clients don’t know to hire me (or the equivalent)–I think they pretty much get their consultants by word of mouth or through some casual search. (I still can’t figure out why the Gore team in the 2000 election hired a statistical consultant who had, as far as I know, never worked seriously on election data before, given that there are so many quantitative political scientists out there. (I don’t actually know who the Bush team hired, but since they won, I guess they got their retrospective money’s worth.))
So, anyway, to get back to the econ professor guy: it’s probably worth clients’ money to hire this guy–I imagine his team has a minimum level of professionalism that’s much better than what’s usually out there. Given the high stakes in many legal cases, and the relative simplicity of the statistical questions that arise, I’m surprised that clients can’t do a better job in finding competent statistical experts.
You're right that their work is better than most of what's out there. This company and its closest competitors are staffed by PhD economists, and provide somewhat more principled work than the total hacks you hear about. The problem is that they are still hired as advocates, and are expected to do some cherry-picking to come up with their answers. This often happens on both sides. Thus the problem isn't finding competent experts, it's finding experts willing to play the game the way it's played.
I've noticed the same thing. With the exception of one very high profile case (Dalkon Shields) the statistical "experts" I've run into usually haven't been statisticians and the opposing lawyers didn't understand the difference. This may have something to do with the number of "statistics" courses that are taught by people without a statistics degree. I would guess that the economist you mention also touts his skill in statistics. I've been told, for example, by a sociologist, that his 4-5 grad school methodology courses was sufficient and he didn't need all that theory, anyway.
The point of an expert witness is not to teach the court science, but rather to convince a judge or jury, or simply scare the other side enough to settle or plea bargain. It's not about being right, it's about winning. This applies to politics, sales, marketing, etc., which is my explanation for why scientists find these kinds of fields so off-putting. For these kinds of jobs, having a newscaster manner (good haircut, nice tan, well cut but conservative suit, firm confident voice, steady body language, etc.) is more important than being a good science teacher, because in the end, it's politics (in the broad sense), not science.
As sad as it is, most people in the world, including most lawyers and judges, have very poor math skills. By this I mean you'd be lucky to find a lawyer in a randomly chosen courtroom who understood algebra well enough to understand compound interest, much less understand the much more subtle notion of variance. So it comes down to a "he-said"/"she-said" and they decide the same way voters do, by how much they trust the speakers, not how much they believe the arguments.
Also keep in mind that not every client could afford your help. My sister and father both work as public defenders and they're not paid enough even for a felony case to afford even a couple hours of a decent scientist's time without losing money.
Finally, a question for Andrew. How do you propose that a lawyer find a statistician as a consultant? Obviously not by asking other lawyers, if your hypothesis that the existing consultants are bad is true. But if you don't know anything about stats, how do you even begin to read through this list of "expert" statistical witnesses?
Bob,
I certainly agree that a consultant with worse statistical skills but better courtroom skills, less shifty eyes, etc., could probably whup my ass in a court case. But what struck me in my consulting is how utterly incompentent some of the statistical work I've ssen on the other side. Some of it is so bad that I actually could shoot it down, I think, even lacking any courtroom skills. And I'm pretty sure these people were paid a lot more than me–my hourly rate is probably higher, but these guys put in a lot of hours.
In answer to your question: I'm not sure how they can find good consultants. That's why those guys who Stephen Dubner wrote about can make so much money–I don't think they have much serious competition. The Gore team in 2000 could've asked around some political science departments, I suppose. I actually think their consultant was irresponsible in taking the job and not referring them to someone with experience analyzing votes. But that's a big problem with asking around–unqulified people are all too willing to take the job.
How about just using Amazon to find them?
One of my old professors helped write a book, "Statistics and the Law" that covers all sorts of varied Bayesian approaches to legal questions. More generally, a search for "statistics" and "law" on amazon leads to dozens of books, of which many are relevant, and one could start by contacting authors of well reviewed texts.
Tuerto,
That sounds like a start. Bruce Levin (coauthor of one of the "Statistics and the Law" books) gets more consulting assignments than he needs (in fact, he's passed on jobs to me). People have contacted me because of the Bayesian book, and they probably contact Sharon Lohr with survey sampling questions all the time. But one difficulty is that the total supply is low. A full-time consultant has 2000 hours/year to play with (assuming no law-firm-type creative billing), but for a professor who's active in research/teaching/writing/, 100 hours/year is on the high end. So not that many total hours to go around for all the potential clients out there.
I think the win at all costs argument makes sense.
And allowing myself to be a tad cynical, I'd say one of the two sides is unlikely to want to hear what a true "statistical expert" has to say. So you may either have to turn away the work or find a way to … um… obfuscate?
Since statistics is not like physics where you can have testably true answers, I wonder what it's like to be an expert witness anyway. Isn't it relatively easy to confuse the jury by pointing out the gray areas? (say a bit of bias here, not sufficient samples there, lurking variables, etc.)
As far as finding a competent statistician: Why not talk to someone at the stat department of a nearby decent university? That won't guarantee genius, but should get someone who knows which n is which and can apply the formulas correctly.
There's a related question about the media, which routinely report statistics without any kind of understanding or analysis at all. Presumably, this is because keeping someone on staff who knows something about statistics doesn't pay–readers either don't notice the errors, or have long since come to expect them.
Albratross,
That's a good idea but I think it's what the Gore team did in 2000–they got a Yale statistics prof who is an excellent statistician but (a) didn't have any experience with election data, and (b) accepted the job anyway.