How to unit test random seeds?

Thread view:

Patricia Shanahan - 20 Dec 2007 00:14 GMT

I have an application that uses two random number generator instances.
The input parameters can specify the seed for either or both of them. If
a seed is not provided in the input, I use a SecureRandom's generateSeed().

I can obviously unit test the processing of specified seeds. The problem
is how to test my use of generateSeed(). Any ideas?

Patricia

Reply to this Message

Mike Schilling - 20 Dec 2007 01:37 GMT

>I have an application that uses two random number generator
>instances.
[quoted text clipped - 6 lines]
> problem
> is how to test my use of generateSeed(). Any ideas?

What are you testing for? Obviously the detailed results will vary,
but perhaps it's worth verifying that that algorithm as a whole works
for, say, 100 random seed values.

Reply to this Message

Patricia Shanahan - 20 Dec 2007 02:05 GMT

>> I have an application that uses two random number generator
>> instances.
[quoted text clipped - 10 lines]
> but perhaps it's worth verifying that that algorithm as a whole works
> for, say, 100 random seed values.

Reply to this Message

Patricia Shanahan - 20 Dec 2007 02:11 GMT

In general, I am unit testing code that processes parameters and manages
default values. Specifically, the question is whether my implementation
uses generateSeed correctly.

There are several conditions that each have a 1 in 2^64 probability,
such as the two seeds, if both defaulted, being equal, or either seed
being zero. I could test for those things and have a vanishingly small
probability of rejecting a correct result.

Patricia

Reply to this Message

Eric Sosman - 20 Dec 2007 02:15 GMT

> I have an application that uses two random number generator instances.
> The input parameters can specify the seed for either or both of them. If
> a seed is not provided in the input, I use a SecureRandom's generateSeed().
>
> I can obviously unit test the processing of specified seeds. The problem
> is how to test my use of generateSeed(). Any ideas?

Idea (maybe not the greatest, and definitely not original):
Some features of a program are amenable to testing, others are
better suited to inspection. If your test proves that "no seed"
does in fact wind up using SecureRandom, just assume SecureRandom
does its job properly.

Signature

Eric Sosman
esosman@ieee-dot-org.invalid

Reply to this Message

Patricia Shanahan - 20 Dec 2007 03:22 GMT

>> I have an application that uses two random number generator instances.
>> The input parameters can specify the seed for either or both of them. If
[quoted text clipped - 9 lines]
> does in fact wind up using SecureRandom, just assume SecureRandom
> does its job properly.

I do assume SecureRandom does its job properly. The question I would
like to test is whether I am using it correctly, as well as the more
general question of how to test one's use of pseudo-random features.

This thread is a last resort before declaring it untestable, at least at
the unit test level, and depending on desk check and system test
instead. At the system test level, I can check questions like whether
repeated runs without specified seeds do behave differently in the
proper ways.

Patricia

Reply to this Message

Daniel Dyer - 20 Dec 2007 11:04 GMT

>>> I have an application that uses two random number generator instances.
>>> The input parameters can specify the seed for either or both of them.
[quoted text clipped - 14 lines]
> like to test is whether I am using it correctly, as well as the more
> general question of how to test one's use of pseudo-random features.

I'm not sure that I am certain what you are asking. Do you want to test
under which circumstances your code makes calls to generateSeed()? In
other words, do you want to assert that generateSeed is called when you
expect it to be called?

Or are you asking how to write tests when you don't know which seed value
your RNGs are using (and therefore can't be certain of the expected
outcomes)?

Dan.

Signature

Daniel Dyer
http://www.uncommons.org

Reply to this Message

Patricia Shanahan - 20 Dec 2007 12:33 GMT

>>>> I have an application that uses two random number generator instances.
>>>> The input parameters can specify the seed for either or both of
[quoted text clipped - 19 lines]
> other words, do you want to assert that generateSeed is called when you
> expect it to be called?

After the call to generateSeed their is also some arithmetic to turn its
byte[] into a long for use by Random's constructor. My missing unit test
is for the process of actually calling generateSeed, converting the
result, and constructing the Random.

> Or are you asking how to write tests when you don't know which seed
> value your RNGs are using (and therefore can't be certain of the
> expected outcomes)?

The easiest solution to that is to not do it. For example, I force both
seeds in my regression tests, so they use the same RNG sequences every
time they run. That works for everything except testing the using of
generateSeed to seed a java.util.Random.

Patricia

Reply to this Message

Daniel Dyer - 20 Dec 2007 13:12 GMT

>>> I do assume SecureRandom does its job properly. The question I would
>>> like to test is whether I am using it correctly, as well as the more
[quoted text clipped - 8 lines]
> is for the process of actually calling generateSeed, converting the
> result, and constructing the Random.

OK, I think I see what your problem is. You want to validate your logic
but can't formulate the assertions because you don't know what
generateSeed is going to return. Is that correct?

Obviously I am unfamiliar with the structure of your code, but the only
thing that comes to mind is to introduce another layer of indirection.
For the RNGs in my Uncommons Maths library, I have a SeedGenerator
strategy interface. This allows me to use seed data from various sources
(/dev/random and random.org, as well as SecureRandom). If you had a
similar arrangement you would be able to substitute your true seed
generator with a mock that returned a known value, purely for the purpose
of testing your arithmetic. You'd have to decide whether the added
complexity is worth it to get the test coverage that you are looking for.

Dan.

Signature

Daniel Dyer
http://www.uncommons.org

Reply to this Message

Patricia Shanahan - 20 Dec 2007 15:40 GMT

>>>> I do assume SecureRandom does its job properly. The question I would
>>>> like to test is whether I am using it correctly, as well as the more
[quoted text clipped - 12 lines]
> but can't formulate the assertions because you don't know what
> generateSeed is going to return. Is that correct?

That's it.

> Obviously I am unfamiliar with the structure of your code, but the only
> thing that comes to mind is to introduce another layer of indirection.
[quoted text clipped - 6 lines]
> added complexity is worth it to get the test coverage that you are
> looking for.

Interesting. That would solve my unit test problem. As you say, there is
a tradeoff between test coverage and layers of indirection.

Thanks,

Patricia

Reply to this Message

Patricia Shanahan - 20 Dec 2007 16:51 GMT

...

>> Obviously I am unfamiliar with the structure of your code, but the
>> only thing that comes to mind is to introduce another layer of
[quoted text clipped - 9 lines]
> Interesting. That would solve my unit test problem. As you say, there is
> a tradeoff between test coverage and layers of indirection.

I thought some more about this, and it can be made very unobtrusive as a
test feature.

I defined a SeedGenerator interface inside my seed-processing class. It
also has a static anonymous inner class object that implements
SeedGenerator using SecureRandom. The seed-processing class has two
constructors, the normal one that defaults to its own SecureRandom
generator, and a package-scope one for testing that allows the caller to
specify the SeedGenerator. I made the interface extremely simple to code
using SecureRandom.

That makes the seed generator selection invisible for normal uses of the
class, but lets its unit test control the seeds completely.

Patricia

Reply to this Message

Arne Vajhøj - 23 Dec 2007 17:52 GMT

> After the call to generateSeed their is also some arithmetic to turn its
> byte[] into a long for use by Random's constructor. My missing unit test
> is for the process of actually calling generateSeed, converting the
> result, and constructing the Random.

If you are willing to use heavy artillery to kill a fly, then you
could run your unit test with byte code instrumentation and
replace the SecureRandom class byte code with a class of yours
that delivered a known result. That way you could test that
SecureRandom is being called and that your "post processing"
of the seed is correct.

Arne

Reply to this Message

Patricia Shanahan - 23 Dec 2007 19:14 GMT

>> After the call to generateSeed their is also some arithmetic to turn its
>> byte[] into a long for use by Random's constructor. My missing unit test
[quoted text clipped - 7 lines]
> SecureRandom is being called and that your "post processing"
> of the seed is correct.

Yes, I can see that is possible. However, based on earlier suggestions
I've reached a compromise that at least keeps me happy. I use a nested
interface SeedGenerator that is defined so that its implementation in
terms of SecureRandom is trivial. The unit test substitutes its own
SeedGenerator implementation that forces known seed values.

There is still a tiny untested code path that your suggestion would
remove, but I don't think it is worth the added complication.

Patricia

Reply to this Message

How to unit test random seeds?

Free Magazines