[FRPythoneers] Doctest

Mike Olson Mike.Olson at fourthought.com
Fri Jul 27 09:14:24 MDT 2001

"S. Luke Jones" wrote:
> Uche Ogbuji wrote:
> > 4Suite has almost 600 test files, making up over 2000 test cases.  It is
> > very rare that it can get through *all* of them without incident.  Often
> > we'll have to tolerate a particular failure in a process of triage.
> Perhaps I'm misreading this. Do you mean you've designed tests that
> you expect to fail, or that you make changes to the code in such a
> way that a great many tests might fail initially?

No, as an example, if we make a change to our XSLT processor that
effects how whitespace is handled, this can sometimes break 100s of test
cases.  We can get a few of them fixed to test that the initial bug we
fixed works, however, we don't always have time to fix all of the other
test cases immediatly.  We want to get the new code checked in because
other people probably need to use it.  So, we either need to allocate
and extra 1+ days for every task to make sure all of the test cases are
running again before we check in (this of course pushes all dependent
tasks off increasingly) or we need a testing system that informs us that
test case XYZ does not pass, but will keep going to perform the rest of
the tests.

> In a subsequent message you invoked your (and another's) Python
> expertise, and clearly it's far superior to my own, so I'm not
> asking about that -- unless it has to do with the reason. On the
> other hand -- i.e., I'm drifting off topic here -- writing test
> tools is an area in which I have at least a modicum of expertise.
> So I'm curious what you are saying here.

It really has to do with the specifics of our project(s).  We deal alot
with XML,HTML, XSLT, etc.  It is often acceptable to have a test case
fail if the programmer knows that it is a white space issue, an
namespace prefix issue, or something in these realms that is known to a
human to be acceptable.  Otherwise we would need _alot_ of logic in our
test comparision functions to account for issues like these.

> The former interpretation is equally troublesome: the whole point
> of -- or at least, a key rationale behind -- unit tests is to allow
> developers to make changes free from the fear that they're sailing
> off into uncharted waters. With a good set of unit tests, it should
> be possible to introduce changes with reasonable confidence that
> things won't break somewhere far away in the source code. See for
> example Fowler's "Refactoring". I also like Kent Beck's thoughts on
> unit testing in for example his first XP book, which I'll summarize
> as: white-box tests should be 100% success; black box maybe not.

I think as mentioned before (and I agree) we are really doing regression
testing, or as I've heard it called use-case testing.  Unit testing on a
XSLT processor (atleast an efficent one) is very difficult.  Almost all
of the components are fairly highly coupled and changes to one cannot be
indivudually tested as you mention.  In fact I would argue that for
almost any large speed critical project. 

To me, unit tests are really development scripts.  They break when you
get into the final stages of project development and the maintence cycle
of a project.  Obviously, if the little chunk of code was working, then
you would not be there fixing bugs which means that the unit test didn't
test the proper aspects, and the rest of the system has been dependent
on a piece of code that was not doing what it was supposed to.  So when
the developer fixes the unit test will break.  If the unit test was what
defined that chunk of codes contract with the rest of the system, then
odds are you just broke the rest of the system.


> --
> Luke Jones = luke/vortex/frii/fullstop/com
> _______________________________________________
> This message sent by the FRPythoneers mailing list.
> Unsubscribe: echo unsubscribe | FRPythoneers-request at lists.community.tummy.com
> URL: http://lists.community.tummy.com/mailman/listinfo/frpythoneers

Mike Olson                                Principal Consultant
mike.olson at fourthought.com                +1 303 583 9900 x 102
Fourthought, Inc.                         http://Fourthought.com 
4735 East Walnut St,                      http://4Suite.org
Boulder, CO 80301-2537, USA
XML strategy, XML tools, knowledge management

More information about the FRPythoneers mailing list