Seth tested his balance every day, sometimes when eating flaxseed oil and sometimes when eating olive oil, and found the following:

This is a pretty graph, and shows that Seth’s balance improved when he ate flaxseed oil and got worse with the olive oil. He conjectures:

A possible explanation is that when the concentration of omega-3 in the blood is low, the omega-3 in cell membranes slowly “evaporates” into the blood. When a cell’s membranes lose omega-3, it doesn’t work as well.

But . . .

As a statistician, my first thought was some sort of measurement bias: Seth knows when he was taking olive oil and when he was taking flaxseed oil, and staying balanced is a tricky enough task that I could well imagine that the results could be affected by his expectations.

Flying blind

I’d be more convinced by a blinded experiment. This is tricky with a self-experiment but it could be done. For example:

1. Get 50 identical vials and pour olive oil into 25 of them and flaxseed oil into the other 25. Label them (e.g., “o” and “f”), then cover up the labels with removable stickers.

2. Mix up the vials in a bag (this is sometimes called “physical randomization” in the sampling literature), then use one vial per day. After use, place them on a shelf in order. Each day, measure your balance and whatever else you want to record.

3. When the experiment is over, peel off the stickers and identify which oil was eaten on which day.

4. If the two oils can be told apart by smell, clip your nose (this might sound weird but actually Seth was already doing this.) If they taste different, mix with some strong bitter flavor (this might mess up Seth’s weight-loss experiment but should be OK for the balance study). If they look different, add food coloring or just use opaque bottles and don’t look inside before drinking.

This simple experiment, with complete randomization, might not capture the time trends Seth is looking for. It would be simple enough to alter the experiment, for example by replacing the vials with larger containers and setting the unit of randomization to be the ten-day period rather than the day. You could even do something trickier, maybe with the assistance of a friend, to set up a pattern with long strings of o’s and f’s without knowing exactly when the switches will occur.

Why Seth’s existing experiment is a good thing: I’m not slamming unblinded studies

I hope Seth (or one of his correspondents) does this randomized experiment. In the meantime, Seth’s results provide a potentially important contribution by motivating new hypotheses. The unblinded experiment was so easy to do (within the context of Seth’s earlier experiments), and placing a requirement such as blinding might have increased the required effort to the extent that Seth might not have gotten around to doing it.

Maybe Seth could make blinding (where possible) a routine part of his future experiments, though. Just as he’s trained himself to perform disciplined self-experiments with precise and regular measurements (something that I never get around to doing when trying out new teaching methods, for example), maybe he could take the next step with blinding.