Sampling rate of human-scaled time series

Bill Harris writes with two interesting questions involving time series analysis:

I used to work in an organization that designed and made signal processing equipment. Antialiasing and windowing of time series was a big deal in performing analysis accurately.

Now I’m in a place where I have to make inferences about human-scaled time series. It has dawned on me that the two are related. I’m not sure we often have data sampled at a rate at least twice the highest frequency present (not just the highest frequency of interest). The only articles I’ve seen about aliasing as applied to social science series are from Hinich or from related works. Box and Jenkins hint at it in section 13.3 of Time Series Analysis, but the analysis seems to be mostly heuristic. Yet I can imagine all sorts of time series subject to similar problems, from analyses of stock prices based on closing prices (mentioned in the latter article) to other economic series measured on a monthly basis to energy usage measured on an hourly or quarter-hourly basis. Certainly some of these signals may be naturally filtered, while others may have energy at higher frequencies.

What do statisticians in the social sciences and economics do to deal with such problems?

Now that I think about this, I see advantages to your stance of repeated regressions at subsequent intervals rather than moving to a full time series analysis. You explicitly give up on reconstructing the “signal” (making a movie of the data) in favor of creating snapshots at various times. You can, I think, still fall prey to aliasing, but you focus on other insights to be gained.

Bill continues:

When you’re done with this question, here’s an idea for a follow-up blog posting: windowing. At least when you’re transforming a time series with a discrete Fourier transform, the assumption is made that the time series is periodic. Because it’s rarely exactly periodic in the real world, the math will distort the signal. Windowing is a way of tapering the time series to zero at both ends, thus moving distortion products out of the band of interest.

That’s an easier problem to fix. While aliasing is uncorrectable after sampling, windowing is done later.

I don’t see attention to windowing in treating time series in the social sciences, either. I’m thinking back through my math to see if I can demonstrate whether the assumption of periodicity applies even if there is no Fourier transform in the picture. Do you see evidence of economists or statisticians applying windows to their time series? If not, why not?

I wonder if this could apply to the fitting of model parameters via MCSim or other tools that offer model parameter estimation for time series analysis. If you have undersampled data and you try to fit a model to that data, even MCMC integration won’t fit the data properly, and so you could get erroneous parameter estimates, or so it would seem. That’s not a problem with MCSim, but it would seem to be a problem with the analysis that prepares the data for MCSim.

I actually don’t have much to say about either of these things. For whatever reason, I’ve avoided time series modeling in most of my work Also, classical time-series analysis hasn’t been so useful for me because that theory tends to focus on direct observations. In my problems I have indirect data. For example, when we’re studying time trends in death penalty support by state, we have sample survey data that gives us estimates for each state and year–and, indeed, we’re fitting time series models to get good estimates–but issues of sampling frequencies seem a bit beside the point.

9 thoughts on “Sampling rate of human-scaled time series

  1. Bill,

    The Nyquist criterion of sampling at twice the highest frequency can be relaxed tremendously when one knows that the underlying signal of interest is sparse (in some basis). The whole theory and associated work belongs to the general (and new) field of compressed sensing (also named compressive sensing or compressive sampling). While the field is new, it has deep connection to older techniques. For instances, when one uses group testing to check for few defects (a sparse set) then the techniques are identical to that used in compressed sensing.

    Cheers,

    Igor.

  2. Try searching for "mixed frequency estimation" and "nowcasting" for some applications in economics such as forecasting quarterly GDP using daily/weekly/monthly data.

  3. Igor and Hal, you've given me much to study (I've already printed off several articles and earmarked a few more). I knew of compressed sensing, but I didn't know much about it, and I didn't really know where to enter the literature on the economics side. Both tips help a lot. Thanks to both of you (and to Andrew for hosting the question).

    (I also know that my statement of the Nyquist-Shannon sampling theorem is incomplete; what really counts is the bandwidth of the signal, not the highest frequency.)

    Is it fair to think that most economic and social science time series analysis does not take advantage of such methodologies? If that be true, do you have any assessment of the magnitude or type of erroneous inferences that are or reasonably could be made from those analyses?

    I recently looked at an undersampled (in the Nyquist sense) time series (unfortunately not something I can talk about yet) in which that data gave seriously erroneous insights. In that case, I fortunately also had integrated data available, and the integration played the role of an anti-aliasing filter.

  4. Bill,

    With regards to compressed sensing, you might want to check some of the tutorials at the Rice site:
    http://dsp.rice.edu/cs
    you might also be interested in online talks:
    http://sites.google.com/site/igorcarron2/csvideos
    (the earlier and probably easier talks are at the bottom of the page).

    The field has been around for a while but crystallized back in 2004 because of some results from Tao, Donoho, Candes and Romberg. However, I am not sure I have seen much in the way of something related to economic or social sciences papers (except maybe when it is realted to portfolio management issues). Probably one of the few reasons might be that compressive sensing works for signals that are known to be sparse a priori, an assumption that might be difficult to hold in these areas. Furthermore, the "sampling" performed in compressed sensing is very much different than the typical sampling procedure in that it is evaluating the signal for different sum of linear combination of the "usual" sample. In other words, at the engineering level, you need to have a way of acquiring this new kind of sample and that means in most cases a substantial change in the hardware acquisition chain. This is the reason why people are looking at developing new types of hardware (http://sites.google.com/site/igorcarron2/compressedsensinghardware ). Some hardware already do this by default like MRI and much of the progress these days in MRI is on changing how we used to sample from these machines.

    How could any of these techniques be used in economics or the social sciences ? I am not a specialist but:

    – If the data of interest is known to you to be sparse in time or space, or both.
    – If there is a way to "incoherently" acquire that signal: for instance if the signal is sparse in time, you need to sample in the Fourier space. If the signal is sparse in terms of labels, you need to be able to sample in groups of labels.

    Then any of the techniques and solvers devised for compressive sensing can be directly used for your purpose.

    As I said before, there are several ways to explain what compressed sensing is depending on your own background. I know by experience that the example of "group testing" used find few defects, provides a nice way for people to make a mental picture of what is going on.

    On the statistics front, the subject is very close to several expressions of LASSO and related algorithms. Emmanuel Candes, Alexandre d'Aspremont, Tony Cai,….. have published on the subject.

    I'll have to ponder on your last question "do you have any assessment of the magnitude or type of erroneous inferences that are or reasonably could be made from those analyses?"

    Hope this helps,

    Igor.

  5. You ask "Is it fair to think that most economic and social science time series analysis does not take advantage of such methodologies?"

    The answer is "yes". Most of time series that economists work with are produced by government agencies and sampled (or maybe "constructed" is a better word) at natural frequencies (days, weeks, months, quarters).

    However, with high frequency data such as financial data, search data, and so on becoming available, there are many interesting questions about how to combine data with different frequencies.

  6. Thanks, Igor. I'll try to get to some of that soon, perhaps this weekend. I was a bit familiar with the Rice effort, as I read about it and their single-pixel camera in my alumni magazine.

    My technical background: originally an electrical engineer doing circuit and systems design, ending up in frequency-domain (swept and FFT-based) test and measurement work. I teach simulation-based system dynamics at a couple of universities, and I'm now doing analysis and evaluation work in the utility field. I use J in my work, and I'm learning R.

    My last question stems from my system dynamics background: while it's sometimes nice to know a confidence / credible interval around a parameter, it may be more important to know if errors / residuals in a model are small enough so that the expected variation is small enough to avoid the need for policy changes. It's perhaps another way of asking if we think the errors from any mis-handling of windowing and aliasing are practically significant in that better inferences would lead to changed policies to be adopted.

  7. Bill,

    I am not sure I am going to answer your question well but let me try anyway :-)

    By acquiring a signal is some "incoherent" fashion, you now have data that represents your signal. A reconstruction step allows you to get the initial signal back for sure but what was discovered back in 2004 is that the new data is smaller in terms of dimensionality. This obvious (simple) compression is useful when trying to make an inference on policy change since the dimensionality of this new data is smaller. However, the noise is itself generally "incoherent" with the signal of interest. What this says is that at a certain level of noise, you will not be able to use this compressed data for reconstruction nor for inference as the noise will have taken over.

    In short, it will probably allow for better inference in that the data reduction process is tremendously simple at the sensor level. However, if noise corrupts too much your data, then the compressed measurements will be disproportionally affected, rendering them useless for inference.

    To get a sense of what I just said, you may want to check the Manifold Signal Processing papers by the folks at Rice (they are in the repository). Look for Mike Wakin, Rich Baraniuk and friends.

    An instance of that is:
    M. B. Wakin, Manifold-Based Signal Recovery and Parameter Estimation from Compressive Measurements.
    http://inside.mines.edu/~mwakin/papers/mbw-sigRec

    also here:

    http://inside.mines.edu/~mwakin/publications.html

    Hope this helps.

    Cheers,

    Igor.

  8. The options are intriguing with high-frequency data, to be sure, Hal, but I suspect someone will need to make it easy to do in order for it to become widespread. Still, in my evaluation work, it'd be good to learn how to do it even the harder ways. More reading and studying is in my future, I predict.

    What gives me pause is not only do most economic and social time series (the ones you mentioned and the ones we read in the news) not deal in the advanced techniques you mentioned; they don't even seem to take advantage of traditional (Nyquist-Shannon) insights. Both aliasing and spectral leakage can become significant problems. We can also get lucky; sometimes we don't have such problems. The problem is that it's hard to know whether we do or not unless the people preparing the data or doing the analysis have taken it into consideration.

    As for CS, Igor, I did some reading on the Rice site, and I'm beginning to catch on at the 10,000 meter level. I suspect the way I'd really get a feel for what it does is to program it, as several of us did for the Analytic Hierarchy Process a few years ago. (That's true both for CS and for mixed frequency estimation and nowcasting.) Maybe I can find something someone's already done in R or learn enough to try it in J (that could be a short program, anyway). I'll have to see if I can find implementation-oriented articles on the sites you've recommended.

  9. Bill,

    You might want to read the Big Picture page I set up as a result of trying to provide a more static page of the field.

    http://sites.google.com/site/igorcarron2/cs

    Of interest are the encoding or measurement matrices being used :

    http://sites.google.com/site/igorcarron2/cs#measu

    and more importantly the solvers needed to reconstruct the (sparse) signals:

    http://sites.google.com/site/igorcarron2/cs#recon

    Most of these solvers are written in Matlab (there should be one in Python) but since the m files in matlab are text file, they are eminently readable. If you want to translate one of these to R or J, I would advise you to look at the algorithms implementing greedy methods as they are the simplest ones to understand. Cosamp and subspace pursuit fall in that category.

    Good luck.

    Cheers,

    Igor.

Comments are closed.