A disaster simulator: roll a month of bad nights

In The costs of badly spending we priced four bad runs, one story at a time. The first was an agent stuck in a loop all night: $18 for a normal run, $540 for the night it looped.

One story makes a bad night look like bad luck. But if you start a few hundred runs a month, a bad night is not luck, it is a rate: some share of runs will loop, and some of those will loop while nobody is looking. The question stops being “what did that night cost?” and becomes “what do the loops cost us a month, and what would a budget per run change?”.

So here is a small simulator. It plays one month of runs twice, from the same dice: once with no limit, once with a budget on each run. Move the sliders, roll another month, and see where the money goes. The numbers are illustrative, the same round figures as the earlier post, not taken from any customer or price list. Put your own in.

How the simulator works

  • A month has a number of runs. A normal run costs around a typical price, some a little more and some a little less (how much they vary is one of the settings), the way agent runs on different tickets do.
  • Each run has a small chance of going into a loop. A looping run keeps spending at a fixed rate an hour (the default, $54 an hour, is the earlier post’s loop: a $0.45 turn every 30 seconds) until someone notices, at some point within the hours you set.
  • With a budget, each run gets its own, as a multiple of the typical run. A run that reaches it is refused its next call and stops. A loop stops early. So does a good run that needed more than its budget, and the simulator counts that run as paid twice: what it spent before it stopped, then the whole run again. That is the pessimistic case, and it is the cost of a budget that is too tight.

The averages are over 200 months rolled with the same settings, so they move when you change a setting and not when you roll.

What to try

Start with the defaults. 300 runs a month at about $18 each, a 2% chance of a loop, noticed within ten hours, and a budget of twice a typical run ($36). With no limit, the average month costs about $7,300, of which about $5,700 is the work itself. The six or so runs that loop are 2% of the runs and about a fifth of the bill. With a budget per run, the average month costs about $6,100: each loop stops at $36, and the seven or so good runs a month that needed more than $36 are paid twice. Roll a few months: the line with no limit climbs in steps, one step per loop, and the line with a budget barely notices them. Note the worst run with a budget: it is not a loop, it is a good run paid twice.

Tighten the budget to about a third above a typical run (1.35×, near the size the earlier post suggested for an agent whose runs are alike). Now about one good run in five hits its budget, and paying those twice eats nearly all of what the budget saves on the loops: about $7,200 against $7,300. A budget that stops the work you wanted is a loss of its own, the second kind of loss the earlier post ends on.

Then go the other way, to three times a typical run. The average month falls to about $6,000: hardly any good run is stopped, and a loop still stops at $54, a single hour of looping. A budget does not need to be tight to work. A loop spends without end, so even a generous budget cuts it off early; what the size decides is how many good runs you stop with it.

Set “Noticed within” to 72 hours, a long weekend. With no limit the average month climbs to about $17,000, and one run can cost close to $4,000. With a budget it does not move: the run stops at its budget whether anyone is watching or not. That is the case a budget is for.

And set it to one hour, a team that watches every run. Now the loops cost little, and the budget costs a little more than it saves (about $6,100 against $5,900), because the good runs it stops cost more than the loops it cuts short. If someone really does look at every run within the hour, a budget per run buys you little. Most teams do not, at night.

What the simulator leaves out

It is a toy, and a pessimistic one in places:

  • A run stopped at its budget is counted as started again from zero. Many runs can be resumed, and with a threshold on the run’s budget a person hears about it before it is reached, while there is still time to give it more.
  • Loops are the only accident. Retry storms, holds nobody released and batches too large for the budget (the other three stories of the earlier post) are not in it.
  • Every run here gets the same budget. In practice you size it from what that kind of run has cost before.

That last point is where to start. You cannot pick the budget’s size without knowing how much your runs vary, and the simulator shows that the size is the whole difference between a budget that pays for itself and one that does not. Run your buckets in reporting mode for a week first: they refuse nothing and record every call, and their history tells you what a typical run costs and how far a good one goes past it. The getting started page sets up a first budget in a few minutes.

The simulator is one small file, disaster-simulator.js, with no dependency. It runs in your browser, sends nothing anywhere and keeps nothing. If you put your own numbers in and get a result that surprises you, write to hello@buckette.co.