The “All Else Equal” Fallacy, again

Here’s the entry from the statistical lexicon:

The “All Else Equal” Fallacy: Assuming that everything else is held constant, even when it’s not gonna be.

My original note about this fallacy came a couple years ago when New York Times columnist John Tierney made the counterintuitive claim (later blogged by Steven Levitt) that driving a car is good for the environment. As I wrote at the time:

These guys are making a classic statistical error, I think, which is to assume that all else is held constant. This is the error that also leads people to misinterpret regression coefficients causally. (See chapters 9 and 10 of our book for discussion of this point.) In this case, the error is to assume that the walker and the driver will be making the same trip. In general, the driver will take longer trips–that’s one of the reasons for having a car, that you can easily take longer trips. Anyway, my point is not to get into a long discussion of transportation pricing, just to point out that this seemingly natural calculation is inappropriate because of its mistaken assumption that you can realistically change one predictor, leaving all the others constant.

I hadn’t thought much about this but then I see that Levitt repeated this error in his new Freakonomics book and on his blog, where he writes:

In SuperFreakonomics, far and away the most common subject of emails is drunk walking vs. drunk driving. In particular, every few days someone writes us to tell us that our analysis is wrong because we are comparing the rate of death per mile driven drunk versus the rate of death per mile walked drunk. Sure, they say, drunk walkers get killed more per mile. But since cars travel much faster, per hour, it is safer to drive drunk than to walk drunk.

It is true that if someone held a gun to your head and said, “If you don’t walk drunk for an hour or drive drunk for an hour, I will shoot you. You choose whether you would walk or drive,” then you might very well want to spend your hour walking drunk. However, in real life, that is virtually never the dilemma you face. Rather, you are drunk in one place and you want to get to another place. The distance you need to cover is what is constant, not the time you will spend traveling.

Thus the per-mile comparison we [Levitt and Dubner] make is the most sensible one.

Well, as the saying goes, saying it don’t make it so. I have to admit that I’ve never been drunk, so I’m not the authority on this topic, but, in general, no, people are not necessarily trying to get from point A to point B. If Point A is 10 miles from Point B, I doubt that many drunk people are going to try to walk from A to B. They might drive from A to B, or they might walk to the bus stop and take the bus, or they might walk to a friend’s house, or whatever. Or, if they drive, they might decide to stop at the supermarket at the way home, adding another couple of miles to their trip.

More to the point, the very existence of drunk driving as an option can put you in the situation where you and you car are 10 miles from home, you’re drunk, and the most convenient option is to get in the car and try to make it back. (Levitt talks about the cost of taking a cab but not about the time cost required to go back and get the car.) If the only available options were to walk or take the bus, maybe people wouldn’t put themselves in this position in the first place–and, in any case, then they could get back home one way or another without having to worry about their car.

In their book, Levitt and Dubner give a very specific example of someone who lives exactly one mile away, but that misses the point, since the real concern about drunk driving is habitual behavior, not this sort of single-play game.

I also think it’s a bit irresponsible for Levitt to mention “other risks associated with driving drunk like getting arrested” without mentioning the risk of hurting or killing someone else who happens to be in the path of your car. (He mentions it briefly in the book, but only briefly.) After all, the danger to others is a key reason that it’s illegal in the first place!

A meta-lesson: Microeconomics ain’t easy, and don’t let a regression–or division by a baseline–be a substitute for clear thought. It’s a classic error to analyze a decision as if it were a one-time choice, without recognizing the underlying incentives that make the situation come up repeatedly. It’s disappointing to see Levitt make this mistake and then see him double down and defend his error.

P.S. The funny thing is that Levitt realizes that many people are bothered by his calculation. He might think a bit harder about why those people might be right! The commenters on Levitt’s blog make a bunch of good points, so maybe he’ll read these and change his mind.

P.P.S. Taking a cab might not be so safe. I’ve known a couple people who were seriously injured in cab rides. Also, cabs sometimes run over and even kill pedestrians, and cab drivers can be drunk too! Once when I was living in Chicago I was in a cab whose driver was extremely drunk–I could smell it. Luckily, he must have realized it, and he was driving very very slowly. Typically I’m impatient when I’m in a cab, but this time I wanted him to take as much time as he needed!

P.P.P.S. Just to clarify, I’m not saying that Levitt’s analysis is necessarily wrong: something useful can definitely be learned from the comparisons of risks per mile of driving and walking, and comparing at different times of day, different states of inebriation, and so forth. Where I think Levitt is wrong is in his categorical claim that his comparison “is the most sensible one,” as if he’s the expert here. Sorry, but I think his blog commenters have more sensible things to say on this than he does.

The real issue, I think, is that, when doing “Freakonomics,” you can be counterintuitive, or you can be sensible, but it’s hard to be both. I mean, sure, sometimes you can be. But there’s a tradeoff, and in this case, Levitt is choosing to push the envelope on counterintuitiveness. As long as nobody actually decides to drive drunk because of the book, it’s fine, I guess. As I said, I think the actual calculations have some value, as long as you’re a bit more careful in the interpretation.

P.P.P.P.S. I’ve been writing about Levitt so much lately, that I should probably state just one more time that I think his blog is great. Really, how many forums are out there where people can discuss this sort of issue. That particular blog entry has 81 comments, most of which make good points. The Freakonomics blog exposes lots of people to rigorous, data-based thinking and gives a wide audience ot a lot of interesting research. And of course I wouldn’t spend my time criticizing (some of) their blog if I didn’t think that it was, on balance, a good thing.

28 thoughts on “The “All Else Equal” Fallacy, again

  1. It's a god example of framing. i read Levitt's post (but not the comments) and didn't even think about his framing of the issue. Clearly, though, it's his framing of the issue that gets him to his conclusion. (I rather suspect, too, that Levitt has himself not been in that situation all that often…or, living in Chicago, his choices wind up being a half-mile walk vs. a half-mile drive/cab ride…the cab ride option being, in Chicago–and in New York–a relatively easy one. Now, in, say, Bloomington, Indiana, now so much.)

  2. Levitt's analysis has some many levels of stupid, it's hard to know where to start. I guess the first thing to mention is that he's just making stuff up:

    If we assume that 1 of every 140 of those miles are walked drunk–the same proportion of miles that are driven drunk–then 307 million miles are walked drunk each year.

    There is simply no factual basis for this assumption. This assumption pre-determines the outcome of the calculation.

  3. I've heard (most likely apocryphally) that after bagel related injuries the second most common emergency room injury in NYC is cab-face, the smooshed and lacerated face from being in an accident in a cab and smashing your face into the divider when not wearing your seat belt.

    I wonder if I'm the only one who marveled that you've never been drunk.

  4. William: One drunk mile walked per person per year…if you assume that half the people in the U.S. get drunk on occasions where they might have to walk, that's 2 miles for each of them. If each is, on average, in this situation twice a month, then that's a 400-foot drunk walk every two weeks. Maybe that works out, I dunno. My objection to Levitt's blog entry wasn't so much that it was obviously wrong but rather that he seemed to be so unjustifiably certain that everybody else was wrong and he was right!

  5. When you cut your hand trying to cut a bagel on the round edge rather than flat on the table.
    Like this (——-)
    not like this:
    (|)
    The former way is much more stable than the latter.

  6. I really don't care how many drunk drivers kill themselves every year. I just don't want to lose another friend or relative due to one of these idiots.

    I doubt if there's been much incidence of drunk walkers killing anyone but themselves.

  7. Following up on Donald, it also would seem that people who are in a position to walk home live in potentially much more dangeroud traffic settings (aka densely populated cities) than those who can only drive. (I'm sure that someone brought this up on the Freakonomics blog, but I've stopped reading the comments there – I don't think they're that good on average).

    I agree that in principle a larger audience for this type of thinking is wonderful, but I'm really concerned about Levitt and Dubner's lack of intellectual humility and the consequences this has for the impact of the blog.

  8. Are drunk walkers and drunk drivers the same people? Especially, are they the same level of drunk? Maybe walkers are more alcohol impaired

    Seems to me that if you know you've got 10 miles to drive home, you might not get as drunk as someone who's just got to walk around the block to get home. Alternatively, if you end up paralytic you might try to struggle home without driving, only to fall under a bus…

  9. He states his motivation at the end:

    "I suspect most people will be surprised to see how close it is to a toss-up"

    Wow. Steve Levitt is a smart, smart guy. And clever too. We need to write it in our principles of economics textbooks that Steve Levitt is a really smart and clever guy.

    Any questions as to why I can't take any of his research seriously? If you're proving 2=1, it could mean you're really smart, but usually it means you're missing something.

  10. Speaking as someone who has not only been drunk, but has been seriously injured while walking drunk, there is counterintuitive and there is just stupid. The car is a complicated, dangerous piece of machinery. You can get seriously injured or killed in a car accident even if you are sober!

    It might be that the calculation being made is the extra chance of getting in an accident added by drunkeness. In other words, driving drunk vs. driving sober as opposed to walking drunk vs. walking sober. Since the chances of getting into an accident while while walking when sober are close to zero, this comparison could well favor driving.

    But from a practical standpoint, its much more relevant to compare the chances of getting into an accident driving while walking. My impression is that driving sober may be more dangerous than walking drunk. Driving could be an activity which you simply shouldn't attempt unless you are 100%.

    As someone pointed out, there is also the ethical issue that drunk drivers put the lives of other people at risk, not so much drunk walkers.

  11. Society protects itself from people drinking and driving because of the collateral damage they inflict on the rest of the population : mass of the vehicle times speed squared.

    Any other modelisation of this problematic without that piece of information is irrelevant.

  12. I agree with Andrew. The information, while greatly misleading, is interesting and has some value. It leads people to think.

    As for Levitt, he has good reason to stand behind his data and not accept it's faults –

    1) He just put out a book and the book is getting a good bit of criticism. Admitting that the information is less than complete/valid in this case brings in to question the rest of the book.

    2) The book is intended to appeal to a mass audience. You need to keep it simple. Most people start to glaze over when you get any deeper than simple math and black and white conclusions.

  13. Drunken state aside, getting around by foot on a Friday or Saturday after dark (most like time for this scenario) is a dangerous thing regardless. As a runner, I try to stay on sidewalks, run during daylight, and run against traffic to protect myself.

    If you wanted to make any statement similar to Leavitt's, you should first study the dangers of being a pedestrian in a city after dark on a Friday or Saturday in general, not the level of drunkeness in that situation. Perhaps a comparison of the danger of walking drunk vs not drunk would be a better study.

    My guess is there wouldn't be a statistically significant difference. The same would not be true for drunk vs not drunk driving.

  14. You write: "More to the point, the very existence of drunk driving as an option can put you in the situation where you and you car are 10 miles from home, you're drunk, and the most convenient option is to get in the car and try to make it back." First, drunk driving *just is* an option; it is a possible mode of behavior, though perhaps one that should ordinarily be rejected. Second, *not getting drunk far from where you want to be* may have been an option–indeed, the most attractive option–available to you at some earlier time. But Levitt was talking about a situation in which that option was no longer available. I'm sure he agrees that it is usually better not to put yourself in that situation–that the choice between driving drunk and walking drunk is an unhappy one (when the third option, staying put, is even less attractive).

    By presenting your remarks as a *criticism of Levitt*, you are guilty of *ignoratio elenchi*.

  15. Andrew's point is really solid. Basing one's decision on death's per mile travelled is really deceptive. Once someone decides to drive to a bar, they are much more likely to go to a bar that is farther away. Levitt is very focused on the "getting home" scenario, but come one, most people decide whether they are going to walk or drive when they leave their house.

    If Levitt really believes that driving vs. walking is "close to a toss-up", he should be up front about it and contrast his findings with say, the odds of a fatality when someone decides to walk vs. drive to/from a bar. The fact that death's per mile travelled is so close basically implies that death's per night out is far less for the walkers. Of course that very much in line with the conventional wisdom, so I guess it's not worth writing, or something.

  16. In response to various comments:

    1. I have no particular pride in never having been drunk–I do, in fact, drink alcohol occasionally. It's just never come up.

    2. I have no idea whether Levitt will do a follow-up post acknowledging his mistakes. It doesn't seem like his style, but who knows? My impression is that he does not take the Freakonomics book and blog as seriously as he takes his academic research. To put it another way, he may have been offering his drunk-driving calculations as food for thought rather than as a serious analysis.

    3. Levitt's analysis is arguably relevant to the particular example in his book (a person who is one mile from home), but he's presenting it as a more general claim about drunk driving and walking, and in any case he is ignoring the repeated-play aspect of the game.

  17. I find it amazing that Levitt is still defending this point, especially in light of the fact that SuperFreakonomics has an entire chapter dedicated to the law of unintended consequences. There's every reason in the world to think that people will use the research of a PhD economist to justify driving drunk (either before or after the fact). That might not be what Levitt expects when he tells people just to take a cab, and that's exactly the point.

  18. Slightly related amusing anecdote about taxi riding as an option: I have a friend who once got arrested for public drunkenness out of the back of a cab. (This was in New Orleans, naturally.) As in the cop saw that she was drunk, and notwithstanding the fact that she took the appropriate behavior in response, actually pulled the cab over to bust her.

  19. I don't think this is necessarily an error, as it does depend on the framing. At what point are we considering making the decision?

    Take an example from my own life. There are a number of restaurants near me that are within easy walking distance for me but far enough that driving would be faster, and pleasanter in cold weather. In the past I have sometimes chosen to walk to one of those restaurants because I planned to have some wine with my meal which ruled out driving home (I also sometimes walk for other reasons, like it being a lovely summer evening). Levitt's analysis, if right, implies that if I want to drink with my meal and I think the risks of drink-driving are too high I should eat at home.

    Or take another example, at a friends' house that is within walking distance, have been drinking, could leave the car there, or could walk home, or could crash at a friends' house for the night. Levitt's analysis pushes me much more to the "crash" option versus walking home, while in the past I would have walked home and picked up the car the next day.

    I also don't understand why you say that the real worry is habitual behaviour. If one of your relatives was killed by a drunk driver, would it really be that much better if you knew that it was only a once-off? Do people whose relatives were hit by drunk teenage drivers feel better about things than people whose relatives were hit by drunk 50-year olds, just because statistically a lot of teenagers grow out of their reckless behaviour while a 50-year old who is still being reckless is far more likely to be reprobate? There might be some difference, but surely it's not very big.

  20. I think it's important to make clear that there's nothing wrong with looking at what would happen if all other things were held equal.

    The problem is how you interpret it.

    If you interpret it as giving you some insight into how things work in isolation from some other things, which can lead to better understandings of the complicated whole system, then this can be very valuable. It can lead to smart ideas, and policies.

    But if you interpret it simplemindedly, as this must mean that if I do X it will result in an increase of Y in the system as a whole, where other things aren't actually held equal, then you can make terrible mistakes, and have very unrealistic misconceptions.

    I have a post on the whole phenomena of simple-minded mechanical thinking. I think it's valuable; it was in Mark Thoma's. It's called, "Scientific does not mean simple-minded", at: http://richardhserlin.blogspot.com/2009/05/scient

  21. Tracy: The reason I focus on habitual behavior is that I seem to recall reading somewhere that it is the habitual drunk drivers who kill the most people. 80% of the damage done by 20% of the offenders, and all that.

  22. Reminds me of some standup comedian I saw once who ended his act with "And remember, drunk driving is dangerous, so if you've been drinking you should drive home as quickly as you can."

  23. Gene: You'll have to do a little research to figure out the answer to this one. Also it's perhaps a good idea to reflect upon the diversity of human experience.

Comments are closed.