MONTE CARLO OR BUST
The technique used most often to help build a statistical picture is called “Monte Carlo Simulation”. I first came across the technique at university studying Plate Tectonics. In that context, a numerical simulation is used to try lots(!) of different arrangements of continents, to try and find one which aligns with data from rock cores on the direction of the Earth’s magnetic field over time, to try and find a set of orientations that agree with all data.
Essentially, Monte Carlo methods are a way of testing a large number of possible results, for a problem which cannot be solved analytically.
In other words, instead of finding an elegant(!) analytical solution for the shape of the statistical distribution of potential future outcomes, we use numerical techniques to identify many different points on that statistical distribution. Each data point is generated by an “iteration” of the numerical model.
HOW MANY ITERATIONS?
Historically, there has been a trade-off between the number of iterations carried out and the length of time taken to run a model, however the goal is to create enough data to produce what appears to be a contiguous set of data.
If one looks at an Ogive, covering a range from 0% to 100%, then something like 1,000 iterations is probably the minimum number required to give an appearance of a contiguous distribution.
But…
With a large number of similar inputs to the model, the output is going to approximate to something like a “normal” distribution. However, the distribution of data points is also going to follow the same distribution. For example: if you have 10 risks, each of which has a 10% chance of occurring, then the likelihood that all 10 risks will occur is:
Probabilty of 10 risks occurring = 0.1^10 = 0.0000000001
In a simulation of 10,000 iterations, there is therefore a 1 in 1 million chance of this value occurring. This highlights that the “tails” of the distribution may not be well sampled, and yet these are often the parts of the simulated distribution that are of most interest (“how bad is the worst case?”).
Fortunately, there is a way of correcting for this:
Latin Hypercube Simulation
Latin Hypercube Simulation is a technique whereby instead of sampling randomly across the whole range of possible outcomes, that range is split into a number of “slices”. During the iterations each slice is then sampled a number of times. If the range is split into 100 slices, and the simulation is run for 1,000 iterations, each slice will be sampled 10 times.
This gives us valuable insight and information to make plans and decisions.
CONVERGENCE
Convergence refers to how quickly a simulation produces a stable solution, that is, one that does not vary significantly from iteration to iteration, normally based around the average value.
If we take our example of 10 risks with a 10% chance of occurring and an Impact of 1, the average value will change very rapidly in the first few iterations.
Across 1,000’s of iterations using Monte Carlo and Latin Hypercube the mean (average) and P80 (see below if you are not sure what this is) values will move during the simulation.
In Uncertainty Management, we use Monte Carlo type methods to build up a statistical picture of the effect of all the various assessed risks. We can do this for whatever objective we’re concerned with, for example, cost, schedule, safety etc.
This builds a picture from the knowledge of the project team and hence may be skewed by any assumptions or biases that the project team may have.
One advantage of using an independent consultant is that as the knowledge is elicited more of the assumptions and biases become evident.
