Ubs writes:

I wonder if I can get your thoughts on sabermetric baseball stats. Basically I’m trying to think about them more intelligently so that my instinctive skepticism can be better grounded in real science. There’s one specific issue I’m focusing on, but also some more general stuff.

(1) First of all, my sense of the sabermetric community is that the big name guys at top inventing the various stats, guys like Nate Silver and Tom Tango, really are smart about statistics and know what they’re doing. Moving down from there, my sense is that among the subworld of SABR-geeks who like to look at the numbers and play fantasy baseball and all that, some of them understand the concepts pretty well and others not so much. My experience in discussion forums is that a lot of the baseball-stats-geeks are enamored with the idea that everything can be measured exactly so they tend to have little patience with uncertainty. These are the guys who will say, “No, you’re wrong, Sal Pecadillo is better than Goober Green. I looked it up on FanGraphs, and Sal’s xFPQAA is 3.582 and Goober’s is only 3.421, so that proves it.” (And then of course all the SABR-geeks are still a step removed from the masses who don’t do sabermetrics at all and are still looking at batting averages and RBIs.)

One thing I worry about is that now that there’s a thriving sabermetric community and the data is easily available, it’s that much easier for someone who *doesn’t* really know what he’s doing to step in and make up a stat and get it “published” whereupon it might get picked up by others who don’t really know either. It seems like they’re inventing some new number every day, and I can’t help wondering if the guy who cooked it up really is statistically informed or if he just crunched a bunch of data and makes claims for his formula that perhaps it can’t support. I guess it’s the same idea as the Internet leading to more typos or factual errors. It’s not that everyone got dumber, it’s just that in the old days there was enough of a hump you had to get over to get published at all that those who did manage to get published were more likely to actually be accurate, whereas now anyone can publish so the inaccurate people are diluting the whole.

I guess that’s not a question, but I’m wondering what you think of that generally.

(2) I’m wondering about the conceptual basis for sabermetric stats. I’m not well educated on statistics, and I probably use a lot of terms wrong. I don’t even really know exactly what “Bayesian” means, but I do get the general idea that it’s a way of looking at probability that says there is some underlying “true” probability that exists, and the closer we look at the data that results from that probability, the closer we can estimate what the underlying probability is. (Is that the right general idea, or am I way off?)

It seems to me that baseball stats have changed in what they are trying to accomplish. They used to be counting stats, but with the new saber guys they’re trying to be Bayesian. In the old days, if you counted up someone’s RBIs, say, there is no probability involved: he really did drive in that many runs. And likewise with batting average: there’s no probability involved, Willie Mays really did get a hit 30.2% of the time.

But it seems to me that the saber guys aren’t trying to measure what *happened*. As a practical matter, they are attempting to predict what will happen next. So, for example, they are interested in how likely it is the guy will get on base in the next at-bat. Therefore they want to look at his OBP, which measures how often he got on base before, but deep down they’re not interested in it as a record of what happened, but only insofar as it is an indicator of what will happen.

But I often sense that it’s more than just prediction. The stats-geeks I encounter in discussion seem to be driven by a desire to measure how good each player is. But they aren’t satisfied by measuring what he has actually achieved (which might be due to “luck”); rather, they want to know how good he “really is”. This is where it starts to feel “Bayesian” to me, as I understand the idea. They are postulating that there is some underlying measure of how good a player that guy is, and all the new formulas they invent are efforts to capture that underlying truth.

Every actual stat is still descriptive, of course. Even if they invent some fancy formula that corrects for ballpark, normalizes against the league average, regresses to some predefined mean, etc, etc, it’s still taking real data of past events and performing some computation on it. But the way they write the formula to define that computation seems to be part of an ongoing quest to find the imaginary number that says how good the guy “really” is.

Hm, I guess that one’s not a question either.

(3) I’m wondering about uncertainty. I know that in polling there’s always a margin for error, but that’s kind of different, because in that case you’re counting things and there’s a real answer that you’re trying to derive from a sample. If there are a million events that can be out or not-out, you could conceivably count them all up and calculate a correct on-base percentage. Or if you don’t have time to compile it, you can count up 1,000 randomly selected ones instead and estimate based on that. That would give you a margin of error of the ordinary kind that I understand.

But it’s not like that in baseball. Some guy we’re looking at actually has only, say, 300 at-bats, and we really do have every one of them and can count them all. But we’re not trying to measure what his actual OBP really was; we’re trying to guess what his chances of getting on-base are for the future at-bats. So how do you measure the uncertainty then? I know it exists. I hear about “small sample size” all the time. If a guy has 1,000 at-bats his past OBP is a better predictor of future OBP than if he has only 20 at-bats. That is pretty intuitive, too.

I know that in this predictive sort of sampling, a large sample is always going to be better than a small sample. What I’m wondering is how we measure the relationship between the sample size and the certainty. Is there any basis for attaching an actual margin of error to an actual number of data points? Could I say that if I have 1,000 at-bats to look at that lets me state a guy’s OBP with a margin of error of +/- X points at confidence level 95%, and then if I have even more at-bats I could make X lower?

The other thing I’m wondering, and we’re finally getting close to my specific example, is whether different formulas yield different levels of certainty. As you know, the SABR guys keep inventing fancier and fancier formulas in their attempt to better ascertain some player’s “true” skill level (which I guess would be your Bayesian “prior”, even though it isn’t really prior). If they look at more different factors to try to achieve that, does that put a greater demand on the data in terms of sample size required to reach the same level of certainty?

OK, here’s my example. There are two pitching stats out there which attempt to improve on ERA. Whereas ERA just measures how many runs the pitcher actually did give up. FIP and tRA attempt to isolate the pitcher’s actual pitching skill from external factors like how well the defense played, ballpark factors, and of course “luck”. The two stats take a similar approach but one is more complicated than the other. FIP categorizes every at-bat into four categories: home run, walk, strike out, or ball in play. It then runs it through a formula that weights each of those according to how many runs that particular event creates on average (which I assume is in turn derived from universal data over the whole league), so that the resulting number is parallel to ERA but presumed a better measure of the pitcher’s actual skill.

tRA does the same thing, but it further subdivides the balls in play to ground balls, fly balls, and line drives, weighting each of those individually. Fans of tRA like to characterize it as “more accurate” than FIP, because it’s more precise in its analysis. My intuition balks at that because it feels like the greater subdivision of results ought to carry with it a greater uncertainty in the weighting factor. Imagine we could subdivide the balls in play with even greater precision. Suppose we had the exact speed, trajectory, and initial position of every ball put in play. We could categorize those as finely as we choose, up to a maximum precision wherein every at-bat is in a single category of one. But if we do that then our weighting factors for those categories are the actual result of that one at-bat, at which point we’re back to measuring what actually happened with all the “luck” it entails, bringing us right back to the ERA that we were trying to avoid in the first place.

My reply: Wow–there’s someone named Tom Tango? That’s a great name, in the Vance Maverick or Larry Lancaster level of greatness, I’d say.

OK, now back to the questions, in order:

1. In any field, you need innovators and you need people to apply the innovations. From that perspective, I don’t think it’s so horrible that some poorly-informed people are obsessing on the latest update to on-base percentage or whatever. From general numeracy considerations, though, I completely agree that the third decimal place can’t be so important. That’s the kind of lexicographic-ordering thinking that I hate, and it appears in so many different contexts (for example, when people want “the best” of anything without thinking about dollar cost, opportunity cost, etc.).

But, if people are going to be innumerate, I’d rather they be innumerate about on-base percentage than about batting average.

2. There’s often a fuzzy line between (a) making inferences, and (b) simply “measuring what happened.” I mean, what’s a “save”? For that matter, what’s a “hit”? Etc. These definitions are constructed to be relevant for inferential questions about players’ abilities and contributions to the team.

One way things are changing is that there’s a ton of raw, raw data–locations of where every ball landed on the field, things like that. In that case, the steps going from raw data to inference are going to be more apparent. With old-fashioned statistics such as batting and fielding averages, it can be easier to fool yourself into thinking of them as pure measurement.

Regarding your question/comment about abilities vs. achievements: this is something I’ve talked about on occasion with my collaborator Hal Stern. To take an extreme example, suppose two teams play in the World Series, and Team A is obviously better, both in the regular season (as measured by won-lost record and maybe also by Bill James-like aggregates of individual statistics) and even in the series (hitting better, pitching better, and scoring many more runs than Team B), but Team B still manages to eke out a 4-3 series victory on the basis of a couple of 5-4 and 3-2 victories achieved by lucky bloopers into center field when there happened to be a guy on second base, etc. I.e., luck. Well, it doesn’t matter, right? Team B is the winner.

In evaluating players, though, you want to factor out the luck, as well as you can–especially if your goal is to evaluate how well the player will perform in future years. So I think it’s important to make your inferential goal clear (which I think Bill James has been pretty good at, in general). This sort of thing might help resolve some of your specific questions about tRA etc.

P.S. I’ve been told that Tom Tango is not his real name. Bummer.