17 replies to this topic

[eddie]

I'm in the process of upgrading from gcc 3.3.4 to 3.4.4, and I'm finding the unit testing I've done invaluable. It's catching memory leaks and just other unsightlies, so hopefully when I'm done my app still runs as normal. :)

Do any of you out there have any opinions on unit testing? Like it? Hate it? Never used it?

Just curious. When I was working in gaming, I found that unit testing was mostly deemed too much work for too little gain, and I'm curious if that attitude pervades the hobbyist coder as well.

(I personally think it's fantastic for the hobbyist. My time is punctuated with lots of stops because of "real life" and work, so writing a failing unit test and making it work divides work into nice time slices to get things done, and feel accomplishment).

[monjardin]

I've never used it in an automated, regression testing form.... usually testing something real quick inline, remove the test code and go on. I want to get into unit testing though.
Are you using a framework like boost::test or CppUnit?

[eddie]

I personally was using boosrt::test at home, and CppUnit at work.

I found that I really liked neither of them. CppUnit is a *mess* internally, and if you try to extend it, you'll find more pain than it's worth.

Boost::Test was sort of nice, but it was a pain to set up tests, and I never found it to be all that easy.

I ended up rolling my own (it actually was only about an hour to do so), but I kept the Boost Program Execution Monitor (that thing rules!). This way I have automated unit testing, memory leak detection, and control over all parts of it.

It's *very* handy. :)

[eddie]

Oh, and just to respond further:

I used to write tests similar to what you've got. Something really quick, asserts that what I get is what I have, and then remove it.

But the more I thought about it, is that it's *valuable* to keep around those assertions in code form, and better yet, to run them against my builds. Right now when a library gets created, it then automatically runs the unit tests afterwards to ensure that what I expect to be the output still happens. If it fails, the build breaks.

Saved my bacon more than once. :)

[monjardin]

What about testing the GUI and graphics? I read an article that suggested separating all the logic and unit testing it, while assuming a widget is going to work properly with the output of the tested code.

The make or break point with a business would be making the tests so easy/quick to put in place that the time spent making test code is worth it in debugging and maintenance time savings. This can be a hard sell on small budget, short timeline projects.

I'm trying to justify the time investment to build a process and go at it. Of course, I don't see how I could implement it efficiently on my microcontrollers =), but the Windows/Linux apps are another story.

At work, we have a body of code that's older than me. It's gone from DG assembly to FORTRAN to C, and we are hoping to modularize it in C++. Automatic regression testing would be a HUGE time savings. Every time they upgrade an operating system or compiler version there is a big effort to work out the kinks that inevitably arise. (At least that’s what I’ve gleaned from my few months here).

Anyway, I'm going to force myself to do unit testing on my next project. It sure would boost my confidence in the code.

Do you think I would be better off starting out with my own home rolled framework, or use one of the dozen so-so solutions out there?

EDIT:
I went back and rechecked this article I had read a while back. I may give CxxTest a shot.

[Altair]

I'm all for unit testing, but I think you have to build your code base from scratch TDD in mind than try to add unit tests to existing code base. I feel it's a bit like exception safety or const correctness, i.e. trying to plug those into an existing code base would be painful experience, but if you have luxury of building code base from scratch (which you have in hobby projects) then it's good way to go.

I have used unit testing in my small hobby project and find it really valuable for writing stable code. It's nice to have those tests around to support refactorization as well, i.e. when you think you got better implementation idea for some component, you got tests around to check if your new implementation does the right thing. I'm not sure how easy it is to test higher level code (e.g. graphics), but you can use it at least for low-level code (e.g. core components such as containers or math classes for instance), which is better than nothing.

I think one very important thing in unit testing "framework" is that it must be very easy to add new tests, which encourages testing. For each new component I test I just basicly write a function and the test code inside it, which gets automatically added to the unit test list, so I don't see any need for external unit testing framework since it's very simple to implement (basicly a simple class + macro). Also you need to design your code to be modular so that each component can be easily ran in isolation, but that's good practice with or without unit testing.

Cheers, Altair

[eddie]

I'll answer a couple of things here.

GUI testing:

Well, I haven't done much, admittedly. The one thing I have done, is a 'chat box', and unit testing was really valuable for that. Basically I did a variant of what you proposed: rather than separate the rendering, I stubbed it, and then made sure the logic did what I thought it should.

I don't know how this will scale to the rest of the graphics stuff, but for GUI with functionality, I found unit testing invaluable. Two techniques had to be used to make it worthwhile however, mocking and stubbing.

Legacy code / large code bases:

I work on a large code base at work now, that's never seen unit testing. We tried using CppUnit to do our testing, and while it works, it's got some kinks.

Putting unit tests into old code *is* quite scary: especially the code we have here, as it's very monolithic, highly coupled, not documented, and noone knows how it works. That said, it's been immensely useful to take the time and wedge out the parts that can be unit tested, and unit test them.

That said, the approach I take is to unit test the sections I'm working on/changing: retrofitting unit testing into it would be a thankless job that would probably kill my affection for unit testing. But when fixing code or implementing new things, I find unit testing to be a higher level 'assert' that I simply can't do without.

Testing libraries:

I recommend you read up and pick one. I read the same article and was about to try CxxTest, but I'm supremely anti-Perl, and I dislike additional build steps than I have to. I realize those are weak reasons, but it's enough for me. :)

Before I rolled my own unit testing library however, I had used Boost::Test and CppUnit. While that doesn't make me an expert, it definitely gives me a perspective on what I need/want in a testing library, and lets me flesh out the custom things that I want.

i.e., one thing that I had a hard time setting up, was making a unit test fail if there was memory allocated during it, but it wasn't unallocated in the course of the unit test itself. This has saved me a score of memory leaks, and was relatively easy to implement in my own unit test library. (CppUnit has a faculty that I could put this in, but their hierarchy is disgusting, and while Boost::Test uses the CRT on windows, there's no such analogue on linux (that I know of, save Valgrind), and even on Windows it won't make your build fail (return non-zero), since it's an atexit() call, I think).

[monjardin]

I downloaded CxxTest and gave it try last night. It only took minutes to get it and going. And to my surprise, there was a Python port of their code generation script. So, I didn't even have to install Perl.
I'm heavily refactoring some old code that I am about to reuse for a new project, and I'm going to implement unit testing with CxxTest while doing it. It's on Windows with VC++ 2003. I just made a sperate makefile project to run the unit testing. I find the scripting facilities in the regular MSVC projects a pain.
Thanks for the inspiration. :w00t:

[eddie]

Heh, good luck man. I'd be interested to hear your findings, or whatever irks you or you like about it. :)

If I can recommend something too: if you can sneak memory leak tests into your unit tests, they're invaluable down the road. I've got a rather neat/cheap trick for it myself.

Cheers!

[Altair]

Something I have wanted to do in unit tests is to measure the performance of some components to make sure changes in a component doesn't have large negative impact in some scenarios. Anyone have good (and simple) suggestions how to do this? You can of course easily profile how long some some piece of code takes to execute, but you need to measure this against something to check if the performance has dropped below a set threshold, and this code is potentially executed on different machines so hardcoded performance values in the code doesn't quite work.

Cheers, Altair

[monjardin]

That's a good idea. Perhaps save previous time values to a file for comparison. Either manually create a file with set initial values, or have the macro code create one if it doesn't already exist.
You could set a tolerance similar to the floating point checks a lot of the testing frameworks have. A precentage change might be more valuable than a fixed time delta.

[eddie]

I tried setting this up myself, but I found it quite difficult.

Besides the reasons you mentioned already Altair, the values of a performance run would vary *huge* depending on what the machine is doing at the time. Even on a machine doing nothing, with it's thread and process priority set at "realtime" (talking Windows here), the variance was too large to apply a meaningful threshold, especially on things that would occur at a very very fine granularity.

If you find something for this, that woudl be great, but all the solutions I dreamed up seemed to have an issue.... Perhaps if you had a realtime OS running the tests, then you could just capture a benchmark on the first build and compare it. <shrugs>

Let me know if you do find something! :)

[monjardin]

Wouldn't a game console qualify as a system without much background noise? For Windows programming, you are absolutely right. I'll still keep it in mind for my realtime work though.

[eddie]

Agreed, if you had it running on said system, that probably would be a good choice.

That said, I do my development on windows/linux now, so I don't know if that's an option. (I don't know if Linux will have the same problems as I had with Windows: would be interesting to find out).

There are a couple of hitches I'll mention however:

1 - Best policy is to have unit tests run as part of your build. If you run it against a devkit, everyone will need to have a devkit, and usually that's not possible, as dev kits are quite expensive.

2 - I'm not sure if all devkits support the ability to run unattended: I know that PS2 and Xbox did, I'm not so sure about GameCube. NextGen stuff, I'm not so sure about... I'll find out soon however. :D


That said, these points are relatively minor.

To combat #1, above, I'd advise at least just having one build machine that builds on code checkin to run the unit tests. It's not as good as every developer having their build fail if they write code that breaks a unit test, but if you can't afford a devkit per seat, this is a nice compromise.

To combat #2, unless you get a *lot* of stub/mock libraries going, chances are you're unit testing code that's cross-platform (I could be wrong), so one devkit would count as all. Even if that's not the case, one platform is better than none. :)

Funny, I spent most this point arguing against myself. :)

[Altair]

eddie said:

Besides the reasons you mentioned already Altair, the values of a performance run would vary *huge* depending on what the machine is doing at the time.

I was thinking of executing the same code large number of times and picking the smallest time.

Cheers, Altair

[monjardin]

That's even better, and it can still be wrapped in a macro that only requires a single line test.
BTW, I caught my first real error with a unit test a few hours ago! It's something I probably wouldn't have noticed until integration too. :)

[eddie]

Altair said:

I was thinking of executing the same code large number of times and picking the smallest time.

Cheers, Altair

Interesting. Let me know how it works out for you, I'd be interested in implementing something similar if it works!

[eddie]

monjardin said:

BTW, I caught my first real error with a unit test a few hours ago! It's something I probably wouldn't have noticed until integration too. :)

Cool! You're well on your way to becoming Test Infected :)