How to replicate random bugs in Ruby

Yesterday, someone Tweeted that a random test run of his code had produced a rare ordering-related bug that he couldn’t replicate because he didn’t know the seed value.

This suggests an interesting dilemma. Randomness is good because it exercises your code more rigorously, enabling you to identify more bugs. On the proverbial other hand, random bugs are tough to fix because they can’t be replicated.

The “obvious” solution — recording the seed when you generate a random one — is infeasible in Ruby because srand()’s return value is not the new seed but the previous seed.

Here are two solutions, both involving recording the seed to enable you to replicate any bug. Whenever you find your code making random calls — and don’t forget things like looping through hashes, whose ordering is not guaranteed — you’ll want to record your seed value:

1) Wrap random calls in begin... rescue... end blocks and make sure your rescue section prints the seed value or saves it to a temp file. I suspect the strange behavior of srand() — its return value is not the new seed but the previous seed — is presumably designed to make this possible:

irb> srand(1)
=> 9664034031542159759956844050608130405
irb> srand(2)
=> 1
irb> srand(3)
=> 2

This seemingly odd behavior enables you to grab the seed that produced the exception just by calling srand() again. This spares you the hassle of saving the seed when you initially call srand() and have not yet hit an exception.

2) An alternative approach — which I prefer because the seed value is really a global variable that should not be entangled with your code — is to set and save a random seed value. This seems impossible, since srand() returns the previous value and randomly selects a new value, which it does not return. But you can call srand() three times, first to generate a seed, second to save the seed in a variable, and third to set the seed. This lets you set and record a random value each time you run your test code:

irb> srand() # generates random seed
=> 230484092778069480813792111804730187148
irb> rand_seed = srand() # stores random seed in variable
=> 286682958024894595375678230594990984042
irb> srand(rand_seed) # sets seed
=> 185843738191625995823601334214935582043
irb> srand(rand_seed) # test: it works!
=> 286682958024894595375678230594990984042
irb> srand(rand_seed) # test again: still working!
=> 286682958024894595375678230594990984042

The first solution is arguably cleaner (aside from its entanglement in your code). But the three calls to srand() can be encapsulated in a function that sets a random seed and returns its value, as follows:

def srand_and_return_seed
  srand()
  rand_seed = srand()
  srand(rand_seed)
  rand_seed
end

seed = srand_and_return_seed

This works well:

irb> seed = srand_and_return_seed
=> 199008925976592588622036659089197447054
irb> seed
=> 199008925976592588622036659089197447054
irb> srand()
=> 199008925976592588622036659089197447054

Either way, you get the benefits of randomness while retaining the ability to replicate even the most obscure bugs.

Posted by James on Tuesday, July 26, 2011