Jacob:

You've chosen a trivial example where your assert can compute the results of
the Square() function you are calling.  That is hardly a typical situation
or there would be no reason for the function to have been created.

double v = getRandomDouble();
double v2 = AccountBalance( v );
assertEquals( v2, ? );

So explain how you get the value to type in ? given you don't know what the
input will be.  Perhaps you would do the computations you read about in the
AccountBalance() method inline to see if those and yours matched?

And how exactly did you come up with v*v as the value to test against?
Did you copy it from the function you're testing?  Do you expect that to
fail?

Did you get somebody else to write the code?  Do you implement all the
code twice, independently and check them against each other?

I fail to see how you are going to automatically test this complicated
logic.

Ben Pope

This is fine.

This is completely broken. You can't test an implementation of 'square'
with an identical implementation. You need a separate representation
for your expected result. Otherwise, you are not testing anything.

This may not ever be revealed using random inputs, but in the case of
'square' this is a moot point. The contract of 'square' must stipulate
that the input (v) is invalid unless
'v * v < "max double"'. Since such inputs are invalid by the contract,
there is no point in testing them.

This is also wrong. The boundaries of the input is stated in the
function's contract. It is not something determined by the user's level
of experience. Your test cases must cover the boundary conditions
stipulated by the function's documented contract *as* *well* *as*
boundary conditions based on white-box knowledge of the function's
implementation. If you cover these cases, plus a small assortment of
well-chosen "sanity" values, you don't need to waste time with large
amounts of random data.

If you can't test your function in this way, it is probably not
factored correctly.

"Properly selected" is fine. If you miss some of those (there may
be MANY remember), the random cases *may* catch them.

That's it. You are not supposed to replace any of the good stuff
you are already doing. It's just a simple tool for making the whole
package even better.

Sure you do.  Unit tests can stop a lot of bugs before they happen and
before tracking them down gets difficult.  The ones that remain mean
that you have to track them down as you normally would, write a test
for the condition that causes the bug to replicate, and then fix your
code until all tests pass.

This means that changes you make to the code later in refactoring or
adding features do not reintroduce bugs you have fixed before.  Think
about how many times you have fixed a bug only for it to turn up later
because of changes you or someone else made to the code.

It helps to write the test first and to write the test independant of
the code in question.  For instance, my latest batch of additions to
our code base involved adding features that were available in a
different code base...one we are depricating.  My tests simply verify
that the same results result from the same inputs since at this time I
want the answers to be the same.  I chose those values randomly but I
put them in as static values in my tests.

Forgetting is also important as I described above in which bugs
reappear after being fixed ages ago because you or someone else forgot
what caused them and put that problem back when altering the code.

The result is a test that works fine in normal cases,

Random inputs are difficult to regenerate.  It might be beneficial to
initially create some random inputs but always put those as static
values in your test.  This may cover some forgotten conditions yet
remain predictable and traceable.  Remember, unit tests should be
completely automatic.

Yes, that is how unit tests should be performed.  The don't test the
code, they test the interface to make sure the code conforms to that
interface and that the interface is what is needed.  They also serve to
document your code base fairly well.

the other way to get coverage is to get same some tests written by
people unfamiliar with the inner workings. The will test things that
"don't need" testing.

There is where TDD comes in.

If we write one test at a time .
  Write Just Enough Code to make the test pass.
    Refactor to improve the current state of the design

We are only writing code for tests we already have. The next test is
only needed if we need to code something or to strengthen the
corner-case tests of the code that we have just made.

This way - there is no forgetting.

To make this achievable, each test case (method) should :
 1) only test one aspect of the code
 2) Have as few asserts as possible (1 being the best)
 3) Be small (like any method) ~ 10(or what ever your favourite number
is) lines of code.
 4) be fast - the faster they run, the more we run them continuosly,
the sooner we find problems.
 5) Do not use/touch: Files, Networks, dbs - these are slow compared to
in memory fake data/objects.

you might want to google 'seeding with time' to see why its not a great
idea....  especially when unit tests are concerned.

white box /black box.... all the same really from a testing PoV...  the
only difference is how tolerable the test case is to the code design
changing. White box..not terribly tolerant. Black box...tolerant.

With TDD, its better to consider the unit tests to be 'Behavior
Specification Tests'.  They are validating that the specified Behavior
exists within the code under test. But each specification test is
specifying a small part of the code under test, as we have multiple
small test cases. Not few large testcases.

For example, we have Calculator class that can Add, Subtract, Multiply &
Divide Integers.

So we'd have the following tests...

testAddingZeros()
testAddingPositiveNumbers()
testAddingNegativeNumbers()
testAddingNegativeWithPositiveNumbers()
testAddingPositiveWithNegativeNumbers();

testDividingByZero()
testDividingPositiveNumberByNegative()
....

I Don't need to have tests for different values within the Integer Range
within each test case, as I have separate testcases for the different
boundaries.  One benefit of having separate named testcases rather than
lumping them all in a single testAdd() method, is that I can write Just
Enought code to make each test pass. However, the biggest benefit comes
later when I or someone else modifies the code and one or two Named
testcase fail rather than a single test case.  Immediately - with having
debug!  I can see what has broken.

"typing....  run all tests ... bang!
 ...
 testAddingNegativeWithPositiveNumbers() failed - expected -10, got -30)
"

I know I've broken the negative with Positive code somehow, but I also
know I Have Not broken any other conditions (testcases).

if all of those asserts were in one testAdd() method, then any asserts
after the one testing -10 + 20 would NOT be run, so I would know if I've
broken anything else.

This might seem like a small thing, but when your application has 1700s
unit tests, its so much easier to see whats happening quickly with this
apporach.

Now each of these test cases my end up being the same apart from the
values passed to the Calc object and the expected output.

In that case I'd do one of two things:
1) refactor the tests to use a private helper method
  private void testWith(Integer num1, Integer num2, Integer expected)..

2) Apply the 'ParameterisedTestcase pattern.

Andrew

Well, no. You still need the unit tests for regression testing purposes.
(Make a change; does the code still obey the contract on it as expressed by
its test regime? If a unit test fails, it means that the code no longer
meets it contract.)

Unit tests are also a really good /development/ aide, if you write the test
cases first. Express your preconditions and postconditions, then write the
code to make the pre- and post- conditions hold true. The test cases are
often easier to write than the code that implements the logic required by
them.

Which is why no test regime is complete if it relies solely on unit-testing.
You want to expend some effort exposing the code to novel inputs -- just to
see what happens. My argument is that these novel inputs do not belong in
/unit/ testing.

See above. No testing regime is complete if it relies solely on unit tests.
By all means, run your code through random inputs if you think it will
discover failures. But do not make it a main feature of your unit test
suite, because a unit test must be 100% repeatable from run to run. (Else
how do you know that you've really fixed any failure you've discovered?)

If other kinds of testing show a failure, by all means add that case to your
unit test suite [when it makes sense] so that it doesn't happen again.

[Unimpressed.] Yes, you *could* do that. But another important feature of a
unit-test suite should be that it is easy to run, not requiring any special
setup. In short, it shouldn't require any parameters, and yet still be 100%
repeatable from run to run. That means hard-coded inputs.

My experience in many different organizations is that the QA teams expect
code to be unit-tested by the developers before being turned over to QA.
Developers writing unit tests means that the unit tests are white-box, of
necessity.

Story time! Consider your reaction to a failing test case.

"Gee, that's odd. The tests passed last time..."

"What's different this time?"

"Well, I just modified the file FooBar.java. The failure must have something
to do with the change I just made there."

"But the test case that is failing is called 'testBamBazzAdd1'. How could a
change to FooBar.java cause that case to fail?"

[Many hours later...]

"There is no possible way that FooBar.java has anything to do with the
failing test case."

"Ohhhh.... you know, we saw a novel input in the test case testBamBazzAdd1.
I wonder how that happened?"

"Well, let's fix the code to account for the novel input..."

[Make some changes, but do not add a new test case. The change doesn't
actually fix the error.]

"Well, that's a relief... the test suite now runs to completion without
error."

These are harried, busy developers working on a codebase that has thousands
of classes, and they're under the gun to get code out the door... they cut
corners here (bad developers!) but I think we can all relate to them.

Random inputs in a unit-test case can:

1. Mislead developers when a failure suddenly appears on novel inputs. If
they aren't working on the piece of code that the random inputs test, they
have to switch gears to understand what's going on;

2. Mislead developers into believing the code is actually fixed, when in
fact it is not, when the failure disappears on the next run of the test
suite.

3. Can create an air of suspicion around the unit-test suite. (To make
errors go away, just run the suite multiple times until you get a run
without errors.)

-- Adam Maass

An observation; not written in stone; a subejective view.

Ignoring TDD, no unit test ever has and no unit test ever will verify a
requirement or testify to completeness of behaviour. You seem to think
that unit testing is to help find all possible inputs for a given
behaviour; I don't think this is true.

Unit tests are regression tests.

When you introduce new feature X in an iteration 5, you write unit tests
to show some confidence that the feature works; you're not guaranteeing
it works for any subset, or for the entire range, of input
possibilities. You could easily have a flaw in the program that gives
the correct output for a given input, but for entirely the wrong reason,
as would be apparent if you used input+1; but you didn't. The unit tests
you write in iteration 5 are, in fact, a cost without a return*.

When you introduce feature Y in iteration 6 is when you see the returns
for your iteration 5 unit tests. As when you run these again, and they
all pass, then you know that whatever you did in iteration 6 didn't
break those parts of iteration 5 that seen to run before. But they still
don't guarantee that feature X is fully tested. If you missed a test in
iteration 5, then re-running the tests in iteration 6 won't help. And
you could still have that bug iteration 5. Unit testing will never
uncover it. All they do is show that whatever you did in iteration 6
didn't change much.