For service in gopher, dial 70.

December 16, 2007

Testing is hard

Filed under: programming — Dave @ 3:14 am

In a software company everyone’s a tester (or should be), which is essential, because no matter how many test cases the professional testers cover they really can’t get in the head of the user. That and testing any significant product is hard.

My main work machine is set to (British) English as my main language, but to display dates, times and currencies in French. Much of the software I use that supports French uses the wrong setting, but it took me two weeks to notice that my fingerprint scanner Reussi!-ed instead of Success!-ing, since as a Canadian, I’m used to just turning the cereal box around whenever there’s French on it. So eventually I’m going to have to switch to Chinese, that should be jarring enough for me to notice.

Lesson #1: There are more edge cases than are dreamt up in your philosophy

For the upcoming m.euri.ca (tools for living my life with my mobile phone), I was playing with the King County transit planner website and accidentally told the system I wouldn’t mind walking 20 miles between stops — putting every stop in walking distance and turning the task of finding the best route into a variation of the travelling salesman problem AND mildly DoS-ing the system:
King County DoS

Lesson #2: Idiot-proofing often underestimates the calibre of idiots out there

I have to be a little vague on this last point, since it involves my software in its vulnerable unfinished and thoroughly pre-beta state. But the basic idea is: no matter how well you’ve tested the components in isolation, actually run through every common scenario.

Lesson #3: If you have a quota of bugs to find, try to do something the salesman said would be easy :)