Saturday, July 24, 2010

A different run (Part 4): Multi-threading and seed

This is the fourth and final part of a series of posts detailing the reasons of having different simulation output when rerunning Monte Carlo simulation models using CB. The other parts can be found here:
Part 1: CB Excel functions are not tied to seed
Part 2: Excel RAND() function in CB models - handle with care
Part 3: Running OptQuest with low-confidence testing at Extreme Speed

In this last post I will cover two other reasons for getting different results from simulation runs. The first one is non-obvious: we have seen that running optimization in multi-threaded Excel environment, like Excel 2007 can result in difference in the numbers from simulation to simulation. We use Excel engine for calculating the values of the forecasts in the spreadsheet model, and some dependencies in the model when coupled with multi-threaded Excel engine results in this issue. A good overview of the multi-threading option for Excel recalcs can be found here.

Workaround: If you notice this issue, check if you have multi-threaded calculations enabled. In Excel 2007, you can find this option at Office Button -> Excel Options -> Advanced -> Formulas -> 'Enable multi-threaded calculation'.

The last one is the obvious one. If you have come this far while debugging why you are getting different results from your spreadsheet each time you run your simulation or optimization model, do not forget to check if you have set the option of using the same sequence of random numbers in the CB 'Run Preferences' dialog. You will find it in the 'Sampling' tab in 'Run Preferences' dialog in CB. Note, you might have used VBA code to alter this option and then have forgotten to reset it.

Still not able to find out the reason? Contact us for support.

Friday, July 16, 2010

A different run (Part 3): Running OptQuest with low-confidence testing at Extreme Speed

This is the third part of a series of posts detailing the reasons of having different simulation output when rerunning Monte Carlo simulation models using CB. The other parts can be found here:
Part 1: CB Excel functions are not tied to seed
Part 2: Excel RAND() function in CB models - handle with care
Part 4: Multi-threading and seed

This post deals with a rather obscure setting which can lead to this situation.  In the Options tab in OptQuest, you will find an option called "Enable low-confidence testing" in the "Advanced Option.." dialog box (as shown below).

As mentioned in the dialog box, this setting improves the optimization time by stopping simulations early if the solution appears to be inferior to the best solution. But this option doesn't quite work as well when the simulation is run in extreme speed. The reason is that, in extreme speed, the random numbers are generated in brusts, and the size of these bursts can vary from simulation to simulation. Naturally, the statistics which gets calculated from the trials can also vary based on these burst size. As an example, let's say that the first time you run an optimization model with this setting enabled, the burst size was 500, and the engine decided to stop early as the solution seemed to be inferior. In the next optimization run, the burst size may be 400, and the trials generated when stopping the simulations might be different.

Workaround: Do not use this option if you want to reproduce optimization results.