« Specifying a prior distribution for a clinical trial | Main | Kaiser Fung on business statistics in practice »
January 25, 2008
Rindskopf’s Rules for Statistical Consulting
Our statistical consulting mini-symposium yesterday was great. I wish we'd been able to video it. There was lively discussion of the connections between statistical consulting and research, and the different aspects of consulting in academic, corporate, and legal environments.
I'll be posting everyone's slides. Here's David Rindskopf's contribution:
Rindskopf’s Rules for Statistical ConsultingSome of these rules are universal, while others apply only in particular situations: Informal academic consulting, formal academic consulting, or professional consulting. Hopefully the context will be apparent for each rule.
Communication with the Client:
(1) In the beginning, mostly (i) listen and (ii) ask questions that guide the discussion.
(2) Your biggest task is to get the client to discuss the research aims clearly; next is design, then measurement, and finally statistical analysis.
(3) Don’t give recommendations until you know what the problem is. Premature evaluation of a consulting situation is a nasty disease with unpleasant consequences.
(4) Don’t believe the client about what the problem is. Example: If the client starts by asking “How do I do a Hotelling’s T?” (or any other procedure), never believe (without strong evidence) that he/she really needs to do a Hotelling’s T.
Exception: If a person stops you in the hall and says “Have you got a minute?” and asks how to do Hotelling’s T, tell them and hope they’ll go away quickly and not be able to find you later. (I’ve had this happen, and if I ask enough questions I inevitably find that it’s the wrong test, answers the wrong question, and is for the wrong type of data.)
Adapting to the Client and His/Her Field
(5) Assess the client’s level of knowledge of measurement, research design, and statistics, and talk at an appropriate level. Make adjustments as you gain more information about your client.
(6) Sometimes the “best” or “right” statistical procedure isn’t really the best for a particular situation. The client may not be able to do a complicated analysis, or understand and write up the results correctly. Journals may reject papers with newer methods (I know it’s hard to believe, but it happens in many substantive journals). In these cases you have to be prepared to do more “traditional” analyses, or use methods that closely approximate the “right” ones. (Turning lemons into lemonade: Use this as an opportunity to write a tutorial for the best journal in their field. The next study can then use this method.) A similar perspective is represented in the report of the APA Task Force on Statistical Significance; see their report: Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
Professionalism (and self-protection)
(7) If you MUST do the right (complicated) analysis, be prepared to do it, write a few tutorial paragraphs on it for the journal (and the client), and write up the results section.
(8) Your goal is to solve your client’s problems, not to criticize. You can gently note issues that might prevent you from giving as complete a solution as desired. Corollary: Your purpose is NOT to show how brilliant you are; keep your ego in check.
Time Estimation, Charging for Your Time, etc.
(9) If a person stops you in the hall and asks if you have a minute, make him/her stand on one leg while asking the question and listening to your answer. If they ask for five minutes, it’s really a half-hour they need (or more).
(10) Corollary: Don’t charge by the job unless you really know what you’re doing or are really desperate. Not only do people (including you) underestimate how long it will take, but (a la Parkinson’s Law) the job will expand to include everything that comes into the client’s mind as the job progresses. If you think you know enough, write down all of the tasks, estimate how much time each will take, and double it. Also let the client know that if they make changes they’ll pay extra (Examples: “Whoops, I left out some data; can you redo the analyses?”, or “Let’s try a crosstab by astrological sign, and favorite lotto number, and...”)
(11) Charge annoying people a higher hourly rate. If you don’t want to work for them at all, charge them twice your usual rate to discourage them from hiring you (at least if they do hire you, you’ll be rewarded well.)
Resourceshttp://www.amstat.org/sections/cnsl/index.html ASA section on consulting
http://www.amstat.org/sections/cnsl/BooksJournals.html Their guide to books and journals on statisticsBoen, J.R. and Zahn, D.A. (1982) The Human Side of Statistical Consulting, Lifetime Learning Publications.
Javier Cabrera and Andrew McDougall. (2002). Statistical Consulting. Springer-Verlag.
Janice Derr. (2000). Statistical Consulting: A Guide to Effective Communication.. Pacific Grove CA: Duxbury Press, 200 pages, ISBN:0-534-36228-1.
Christopher Chatfield (1988). Problem solving: A statistician's guide, Chapman & Hall.
Taplin R.H. (2003). Teaching statistical consulting before statistical methodology. Australian & New Zealand Journal of Statistics, Volume 45, Number 2, June 2003, 141-152. Contains a good reference list on statistical consulting.
Posted by Andrew at January 25, 2008 5:29 PM
Trackback Pings
TrackBack URL for this entry:
http://www.stat.columbia.edu/~cook/movabletype/mt-tb.cgi/1404
Comments
Another reference I've found useful is
Gerald van Belle (2002). Statistical Rules of Thumb.
Chapter 8 is on consulting and is very much in-line with Dr. Rindskopf's advice.
P.S. I also like the "Doorknob Corollary": any questions or comments made while the client's hand is actually on the doorknob leaving the meeting will invariably be the most important/difficult piece of the analysis.
Posted by: John Chandler-Pepelnjak at January 25, 2008 6:02 PM.
"Don’t believe the client about what the problem is."
LOL, this is a general rule for all kind of consulting!
Posted by: Jean-Luc at January 26, 2008 1:02 AM.
Very useful. It strikes me that a number of these rules are also good advice for when a fifth-year Ph.D. student you've never seen before shows up in your office and says "I was wondering if I could ask you for some help with my dissertation..."
Posted by: C. Zorn at January 26, 2008 12:31 PM.
A FEW WORDS ABOUT DISSERTATIONS AND DISTANCE LEARNING
The most rigorous part of the dissertation includes the
Methods Section
Study Design
Research questions and hypothesis formulation
Development of instrumentation
Describing the independent and dependent variables
Writing the data analysis plan
Performing a Power Analysis to justify the sample size and writing about it
Results Section
Performing the Data Analysis
Understanding the analysis results
Reporting the results.
When you enter this phase of the program, you are nearing the end of the journey. Given the difficulty of this phase, one often wishes they had previewed what was to come.
Many Ph.D candidates seem to hit a brick wall and feel disarmed when called upon to work on the methods and results section of their dissertation.
This is the point where many students diligently search for help calling on their advisor, peers, university assistance and even Google.
This is also the time when the student asks themselves the question" HOW MUCH HELP IS TOO MUCH".
Surely no one will deny that having your dissertation written for you is very wrong.
On the other hand, it is not unusual for doctoral students to get help on specific aspects of their dissertation.(e.g. APA formating and editing) It also is not unusual for advisors to encourage students to seek outside help.
If you are a distance learning student it is almost essential you seek outside assistance for the methods and results section of your dissertation. The very nature of distance learning suggest the need for not only outside help but help from someone gifted in explaining highly technical concepts in understandible language by telephone and e-mail.
Distance learning, and the avaiability of programs, has increased exponetially over the last few years with some of the most respected institutions (Columbia University, Engineering; Boston University and others) offering a Ph.D in a variety of fields. If you are enrolled in a distance learning program, or considering one, you will be interested in reviewing the reference sites listed at the bottom of this page.
As stated above, many students hit their dissertation "brick wall" when they encounter the statistics section. Frequently, a student will struggle for months with that section before they seek a consultant to help them. This often leads to additional tuition costs and missed graduation dates.
If I were to name a single reason why a PhD candidate gets off track in their program it is the statistics and their fear of statistics.
So, the question is whether or not it is ethical to get help at all. If so, how much help is too much.
I don't know if there has ever been a survey of dissertation committee members who were asked this question, however, I know many advisors take the following position when they suggest or approve outside help:
To a large extent the process is self controlling. If the student relies too much on a consultant, the product may look good, however, the student will be unable to defend his/her dissertation.
It takes a committed effort on the part of the student and the consultant (resulting in a collaborative/teaching exchange) to have the student responsible for the data and thoroughly understand the statistics. The day the student walks in front of the committee to defend, there should be no question as to his/her understanding of statistics.
When their defense is successful, the question of "was the help too much" is answered.
If you are a Ph.D candidate and would like additional information, you may email me at:
Boyd
boyd67@comcast.net
Reference sites:
http://www.usdla.org/
http://www.cgsnet.org/
http://www.statisticallysignificantconsulting.com/
Posted by: Boyd at February 5, 2008 12:50 PM.