18 thoughts on “Debugging”

1. There is a huge amount of work that gone into developing techniques for writing software that works. This includes using languages that support debugging, version control software to manage the software development (especially important when several people are developing the software simultaneously), inclusion of special code to verify correctness of the running code (often running to more than half of the code even though it is never run during production), test suites that are used every time new code is run to assure that things haven't been broken, etc. We have been using many of these ideas in our project.

   I recommend the Software Carpentry program that was presented a few years ago at the AAAS. The mp3s of the lectures as well as the charts may be downloaded from the following site:

   http://osl.iu.edu/%7Elums/swc/

2. Read the documentation.

   More importantly, re-read the documentation and check what the parameters mean. For example, twice I've been stuck on long debugging stretches merely because I forgot that the *exp family of functions in R uses rate parameterization. Just not intuitive to me, so while the syntax for rexp(n,.1) is fine, it sure wasn't doing what I thought it was doing.

3. I've found that the R-help forums are legendary for the rude poster(s). As an additional point, with numerical solutions it can often be the case that neither the programmer nor the program are wrong per se; understanding the approximations that your software uses is critical in debugging, so that, for instance, you don't pull your hair out trying to figure out why sometimes for extreme values pnorm() in R yields NA but not others.

4. or just learn from people that already debugging for decades, software engineers i mean

5. I fully agree with Bill Jefferys. Version control software such as subversion will help a lot with keeping control over what you develop, and tracking down what changed.
   And if your language supports it, write tests! Testing, in my opinion, would be the "Apply the scientific method": build your experiments as tests, and check that your model passes the experiment… Automated testing has changed my life as a developer (for the better): when you add new elements to your model, first check that your tests still work, and you know if you have broken anything – and if yes, what you have broken.
   I had to go back to Excel development recently, and the impossibility to write tests was my biggest frustration.

6. Good advice about asking for help. But, no kidding, "The friendliness of these forums varies."

   IME, just the process of writing up a draft email that is well-organized and detailed enough to pass muster against some of the more … robustly direct … posters on r-help substantially raises my chances of figuring out the solution myself.

7. Hi,

   I'm excited to read this book. It sounds like it would be very useful. One comment though, you write
   putting print lines inside code to figure out where the problem was

   As a computer scientist I would suggest looking into modern debugging software: I couldn't live with conditional breakpoints anymore …

8. Despite the occasional unfriendly comment, R-help is unsurpassed in the range of expertise that can be applied to your problem for free. You get some of the most knowledgeable people in the world to help you, and you don't have to pay a cent.

   You have to remember that R-help is populated with academics, who are happy to point out errors in your thinking – don't take it personally, just use it as an opportunity to improve yourself. If you think you're correct, stick to your guns and provide some evidence and you'll usually win out.

9. I agree with Mathias: the biggest thing you can do to help is create automated unit test harnesses that run tests for you automatically. Unit tests check the smallest unit of functionality presented to users on a function-by-function basis. It's important to check the boundary conditions.

   I'm assuming everyone's using version control — if you're not, you have bigger problems than debugging. How do you run on someone else's machine? Move between home and work? Back out changes if you go down the wrong path?

   For statistical model fitting with something like Gibbs samplers, unit tests require supplying your own random number generator as an argument so you can make the code do the same thing twice in a row.

   For statistical models, you also want a range of regression (pun intended) tests, which test larger bits of functionality, such as model-fitting accuracy. This is what Andrew's always talking about in his blog, papers, and books, so you should already all bee doing this as part of your model fitting. The point is to automate the tests.

   Writing unit tests means thinking about the specification, because you need to know what to test. Unit tests are also very helpful as demos in how the code should work for others.

   I think Bill Jeffreys is suggesting something different: write assert-like statements in the code to check that everything's on track. I would strongly recommend *not* doing that. Internally, things only get screwed up by programming mistakes, and we don't need to check for things that we have control over.

   What you *do* need to test is that public functions get the right kinds of arguments and throw exceptions or return error codes to let the user know. Private stuff should just work if all the params are well formed; adding all kinds of arg checks is not helpful in the long run, as it just clutters up the code. The problem is where to add such tests. The answer in most professional code is nowhere (though opinions do vary on this).

   What incremental means in software is bottom-up development. Build each thing you do on top of well-tested foundations. You can design top down, but development really has to go bottom up. Bottom-up development also helps with one of the fundamental tenets of software design: don't repeat yourself (i.e. don't cut and paste).

   As to the other points, let the compiler check your syntax (unless you're using Perl, where my advice is get a more strongly typed, syntactically constrained language). Use reasonable naming conventions so you won't get confused when you go back and look at variables months late.

   With things like contributed R code, do worry that pieces other people wrote don't work. Usually the operating system or the C compiler will do the right thing. I don't have enough experience with R. Whenever you use someone else's code, check it with simple cases to the extent possible. For an amusing read about when it was the system's problem, read Josh Bloch's blog on Nearly All Binary Searches are Broken. Josh was one of the main Java developers.

   I'd strongly recommend Hunt and Thomas's Pragmatic Programmer for great advice on programming that applies no matter what you're developing.

10. A bit more on version control software.

    It can also be used on any other project where revisions are being made. Consider a paper, developed by one or more authors. You can put the source under version control, which allows the co-authors to make revisions in a way that allows reversions to any earlier version in an easy way, should one co-author's revisions not be suitable.

11. Bob Carpenter wrote:

    —-

    I think Bill Jeffreys is suggesting something different: write assert-like statements in the code to check that everything's on track. I would strongly recommend *not* doing that. Internally, things only get screwed up by programming mistakes, and we don't need to check for things that we have control over.

    —-

    No, that is not what I am suggesting.

    The Software Carpentry project discusses how to do this, and it is along the lines that you describe in your excellent contribution.

12. Bob,

    I hear what you're sayin (also in your comment to the previous blog entry), but what ya gotta remember is, statisticians don't know any of this stuff. I'm on the computational side of statisticians, and I can only guess about what you're talking about. So Murrell's book has to be a good start. But I'll take a look at these other references. The challenge is to take these great ideas that are routine to you, and to put them into the statistical practice of someone like me who does hundreds of statistical analyses per year but never has time to implement version control, "the debugger," etc.

13. Andrew,

    you don't have to implement the entire range of ideas Bob suggests. Only a subset of them will already be a big advance. I'm teaching a course with the Gelman and Hill book (which is really a great book), and I often feel that the book would have served students and me better if the R code was a bit better organized (e.g., some code is broken, does not work as intended, parts of an R file require running commands embedded in other R files).

    Even using Sweave to write the G&H book would have made life much easier for users (I know you did not intend to have people actually run the code, but there are probably a lot of people like me who do run it :-).

    One of the other key reasons for the broken code in the G&H book is the absence of version control; old functions or commands that were changed in file X (or on the command line even, and never saved) are used in their old form in file Y, which breaks the code.

    I would have a local system admin setup subversion, and once it's installed it's really a matter of a simple checkout-edit-commit cycle (with occasional conflict resolution, which also takes almost no time but does occasionally force you to actually talk to co-developers or co-authors, not always a bad thing), which takes no energy or time at all, but brings huge benefits. The lme4 community interacts with the latest lme4 code using subversion too (the R-Forge).

    Fixing these small problems does not mean you have to become a Bob Carpenter, so to speak. These are really very small changes that have a huge impact on the end product.

14. On technical help lists and friendliness or rudeness:

    I am only a very occasional R user and I don't use the R help list, but there are good reasons for me to look at it from time to time. I do follow another statistical list very closely and have participated in several debates on these issues.

    "[V]ery rude" to me means (e.g.) a torrent of crude language or abuse directed at the person, perhaps to the effect that they are lazy, stupid, etc. I don't see evidence of either, even though there are lots of pretty dumb questions, even down to the level of "I am having problems with the foobar package. Can anyone help?", which might tempt people to splenetic responses.

    The R help list is a very popular, very successful list. The standard of expertise shown is spectacular. The posting guide is packed with excellent advice. A fairly large fraction of the postings ignore that advice. Poor questions all add to the burden without helping anyone, least of all the poster.

    This seems to me generic to all technical lists that I've ever heard of that are not selective about membership. In statistical computing, most of the users aren't very interested in either statistics or computing, and many want to get whatever help they can lay their hands on, without caring much about working out a solution for themselves. (We are like that about some matters.)

    The problem is a political question, not a technical one. The question is what to do when people, for whatever reason, do not follow the standards laid down for proper behaviour in a group, a discussion list, and one that they willingly join.

    One solution is to move to full moderation of a list, so that poor questions are just filtered out. That is pretty unattractive for several obvious reasons on any list with dozens of messages daily.

    The solution of most list members is that poor questions should just be ignored. The main advantage of that is that people restrain themselves from saying things that others might think rude or unhelpful. Indirectly it might help in so far as people being ignored might re-post with a better question, or just go away.

    The solution being complained about is that some people — usually "senior" people on a list with recognisable expertise — are very firm in reminding posters of poor questions about the need to be much more precise about what their difficulty is, to read the documentation, etc. As this advice is very much part of the guidelines that people are asked to follow, it seems disingenuous, if not hypocritical, to complain when those people are trying their level best to maintain the standards of the list, exactly as advertised. On the R help list, the advice is very often coupled with whatever constructive comments are possible.

    Of course, being nice is always better than being nasty, too, but being nice alone has not solved these problems.

15. Andrew: I think it's because most statisticians (even computational leaning ones) have very little knowledge of good programming practice (as opposed to theoretical computer science stuff). This is where I think most grad programs let down their students – to do data analysis these days, you really need good programming skills, and most statistical computing classes focus on the technical side rather than the practical side.

16. Nick wrote:

    Of course, being nice is always better than being nasty, too, but being nice alone has not solved these problems.

    The thing is that being nasty doesn't solve the problem either. I can imagine having a fixed type of answer that can be inserted (see p. so and so of such and such manual), along with a polite request to please consult the posting guide. That's about as effective (or ineffective) as telling them to read p. such and such as well as insulting some poor poster who hasn't yet experienced the full level of dysfunctionality academics are capable of.

    I see the same kind of stuff occasionally in journal reviews, another place where people pride themselves on quality control. Nasty, personally insulting reviews. I recently heard a prominent (and very capable) linguist say that she doesn't submit to journals any more (in linguistics one can get away with this) simply because the reviewers are so nasty (this happens a lot more than in areas like psychology, I hear). There's no excuse for journal reviews either. It's easy to reject a paper, utterly and totally, without telling the authors how stupid you think they are.

17. Hehe I remember my blame the computer phase. Only once in ten years of coding has it been the compilier. I bumped into an obscure Vstudio "known issue" where it wasn't handling a flag properly. Took me two days to figure it out.

    My only addition to the list that seems to nab my grad student peers alot is an unwillingness to inspect their data structures. This can be betweenfunctions and in debug sessions for your functions. It can be with your live data or your canned testing data. It can be sanity checks looking for glaring inconsistancies or it can be checks for subtle failures.

    You would be surprised how often I help my peers just by inspecting their variables for sanity, and its not on your list.

    I miss visual studios debugger when I debug in R =/

    One of my great hopes for R is the debugging support in a popular and easy to use GUI will get more elaborate. Seems kind of 1980s to start stepping in a function and check the same variables over and over by hand.

18. It looks to me like Paul Murrell's debugging advice covers 5 of the 9 rules I list in my book, Debugging. Several of his I combine into more general principles, but there are important ones not shown like Make It Fail, Quit Thinking and Look, and Keep an Audit Trail. (The list is on a free downloadable poster with cartoon bugs hiding all over it, at debuggingrules.com). I tried hard to extract the essential rules of debugging, which means you need to follow them all. And I'm pretty sure they're universal, since I show how they apply to software, hardware, plumbing, cars, human bodies, etc…

Comments are closed.