How to replicate random bugs in Ruby
Yesterday, someone Tweeted that a random test run of his code had produced a rare ordering-related bug that he couldn’t replicate because he didn’t know the seed value.
This suggests an interesting dilemma. Randomness is good because it exercises your code more rigorously, enabling you to identify more bugs. On the proverbial other hand, random bugs are tough to fix because they can’t be replicated.
The “obvious” solution — recording the seed when you generate a random one — is infeasible in Ruby because
srand()’s return value is not the new seed but the previous seed.
Here are two solutions, both involving recording the seed to enable you to replicate any bug. Whenever you find your code making random calls — and don’t forget things like looping through hashes, whose ordering is not guaranteed — you’ll want to record your seed value:
1) Wrap random calls in
begin... rescue... end blocks and make sure your rescue section prints the seed value or saves it to a temp file. I suspect the strange behavior of
srand() — its return value is not the new seed but the previous seed — is presumably designed to make this possible:
irb> srand(1) => 9664034031542159759956844050608130405 irb> srand(2) => 1 irb> srand(3) => 2
This seemingly odd behavior enables you to grab the seed that produced the exception just by calling
srand() again. This spares you the hassle of saving the seed when you initially call
srand() and have not yet hit an exception.
2) An alternative approach — which I prefer because the seed value is really a global variable that should not be entangled with your code — is to set and save a random seed value. This seems impossible, since
srand() returns the previous value and randomly selects a new value, which it does not return. But you can call
srand() three times, first to generate a seed, second to save the seed in a variable, and third to set the seed. This lets you set and record a random value each time you run your test code:
irb> srand() # generates random seed => 230484092778069480813792111804730187148 irb> rand_seed = srand() # stores random seed in variable => 286682958024894595375678230594990984042 irb> srand(rand_seed) # sets seed => 185843738191625995823601334214935582043 irb> srand(rand_seed) # test: it works! => 286682958024894595375678230594990984042 irb> srand(rand_seed) # test again: still working! => 286682958024894595375678230594990984042
The first solution is arguably cleaner (aside from its entanglement in your code). But the three calls to
srand() can be encapsulated in a function that sets a random seed and returns its value, as follows:
def srand_and_return_seed srand() rand_seed = srand() srand(rand_seed) rand_seed end seed = srand_and_return_seed
This works well:
irb> seed = srand_and_return_seed => 199008925976592588622036659089197447054 irb> seed => 199008925976592588622036659089197447054 irb> srand() => 199008925976592588622036659089197447054
Either way, you get the benefits of randomness while retaining the ability to replicate even the most obscure bugs.
Posted by James on Tuesday, July 26, 2011