S. Luke Jones
luke at frii.com
Thu Jul 26 21:22:36 MDT 2001
Uche Ogbuji wrote:
> 4Suite has almost 600 test files, making up over 2000 test cases. It is
> very rare that it can get through *all* of them without incident. Often
> we'll have to tolerate a particular failure in a process of triage.
Perhaps I'm misreading this. Do you mean you've designed tests that
you expect to fail, or that you make changes to the code in such a
way that a great many tests might fail initially?
In a subsequent message you invoked your (and another's) Python
expertise, and clearly it's far superior to my own, so I'm not
asking about that -- unless it has to do with the reason. On the
other hand -- i.e., I'm drifting off topic here -- writing test
tools is an area in which I have at least a modicum of expertise.
So I'm curious what you are saying here.
If it is the former (expect failures) then is that by design?
If so, why would you design a test suite that cannot be run in
an automated fashion. This would result in one or more of:
(a) an excuse not to run them ("just this time since we know
we didn't change much").
(b) an opportunity for pilot error ("oops! I didn't see that
"F" in the log there. I thought it was "S").
(c) an unnecessarily lengthy test cycle (which leads to "a").
The former interpretation is equally troublesome: the whole point
of -- or at least, a key rationale behind -- unit tests is to allow
developers to make changes free from the fear that they're sailing
off into uncharted waters. With a good set of unit tests, it should
be possible to introduce changes with reasonable confidence that
things won't break somewhere far away in the source code. See for
example Fowler's "Refactoring". I also like Kent Beck's thoughts on
unit testing in for example his first XP book, which I'll summarize
as: white-box tests should be 100% success; black box maybe not.
Luke Jones = luke/vortex/frii/fullstop/com
More information about the FRPythoneers