The opportunity cost of badly spending

Our last post put a price on four bad runs: the loop that ran all night, the retry storm, the hold nobody released, the batch that did not fit. Each one had a bill, and a budget around the run made the bill smaller.

The bill is only half of it. A budget is finite: a pool of GPU hours, a monthly allowance of tokens, an hourly rate limit on a paid API. Whatever the wrong run spent, the right run could not. That second cost appears on no invoice. It shows up as a release that shipped late, a customer who was told “try again later”, an experiment that was planned and never started. Economists call it the opportunity cost, and it is often larger than the bill.

This post walks through three forms it takes, and what buckette does about each today. As before, the numbers are illustrative: round sizes and prices chosen to be plausible, not taken from any customer. The arithmetic is the point.

1. The work that waited

A team shares one model budget between everything its agents do: 2 million tokens an hour, refilled on the hour. Work that does not fit is queued rather than refused, which is the right instinct.

At 8 am someone queues a backfill: 600 jobs of 50,000 tokens, 30 million tokens in all, fifteen hours of the pool. At 9 am the release pipeline queues its review step, 200,000 tokens, which a person is waiting on.

  • First come, first served: the review runs after the backfill, around midnight. Fifteen hours of a release waiting, for a step that needed a tenth of one hour’s budget.
  • The backfill did not need to finish today. Nobody would have noticed it finishing an hour later.

Nothing was overspent. The pool was fully used, every token on something that eventually mattered. The cost is entirely in the order.

What buckette does. Queue both kinds of work as claims, and give them a priority. A queued claim is served by clrPriority, highest first, and in arrival order within a priority:

POST /claims
{"clrStrategy": {"tag": "AllOf"},
 "clrMode": {"tag": "Async",
   "contents": {"acMaxDelaySecs": 7200,
                "acAutoConsume": true}},
 "clrPriority": 10,
 "clrTargets": [
   {"ctBucketId": "<pool>",
    "ctAmounts": {"tokens": 200000}}]}

The review now waits for the next refill at most, not for the backfill. A claim can carry a callback URL, so the pipeline hears when it is granted instead of polling. And when something is still waiting, GET /claims/<id> says where it stands: its place in line (ciQueuePosition) and why it waits (ciWaitReason: short of budget, behind a larger claim, about to be granted).

Two details matter here. By default a claim that cannot be served yet holds its place, so a large claim is not starved forever by a stream of small ones; a claim marked QueueOpportunistic lets others go first instead. And queued plain consumes on a bucket are served before any queued claim on it, whatever the claim’s priority: to have work take its turn by priority, send it as a claim (a claim with one target is a single-bucket consume).

2. The valuable work that was refused

A support assistant and an internal summariser share a daily allowance of 5 million tokens. The summariser runs every few minutes over chat logs and is worth little on any one run. The assistant answers customers, and one conversation that afternoon is a renewal: a customer worth $40,000 a year asking a question before signing.

  • By 3 pm the summariser has used the day’s allowance, a few cents of value per run.
  • The assistant’s next call is refused. The customer gets an apology and a ticket number; the renewal waits for a human on Monday.

The tokens that would have answered cost about a dollar. What the refusal risked is out of all proportion to that, and the budget, seen alone, cannot tell the two calls apart: a token is a token.

What buckette does, in two parts.

Protect the high-value class. Give the two kinds of work consume classes on the allowance, and put a floor under the summariser’s class: part of the balance it cannot touch, released gradually as the reset approaches so nothing is wasted at the end of the day. The assistant can always reach the floor; the summariser stops above it. Our first post shows the configuration, and buckette can suggest the floors from your own history before you set any.

Record what each claim is worth. A claim can carry a value in units you name, one or several:

"clrValue": {"revenue": 40000,
             "customers": 1}

buckette checks its shape, keeps it on the claim, and writes it as value on every event the claim produces: the grant, the wait, the expiry, and the refusal (bucket.claim_shortfall). It never acts on it: priority still orders the queue, and a value is not a priority. What it gives you is the record you need to see this cost at all. Read a budget’s refusals with GET /buckets/<id>/events?event_type=bucket.claim_shortfall, add up their value, and you have, in your own units, what that budget turned away. A report of consumption and refusals by value is planned; the field ships first so that the history is already there when it arrives.

Leave the value out when you do not know it. Absent means “unvalued”, not zero, so a missing figure never reads as worthless work.

3. The experiment that never ran

A research group rents 100 GPU hours a day and splits them evenly between four teams: 25 hours each, every day, whatever happens.

  • On Tuesday, team A has a sweep that needs 60 hours and could change the quarter’s model choice. It gets 25, so it waits until Friday, or is cut to a third of its size.
  • On the same day, teams C and D have nothing ready. Thirty of their hours sit idle: about $75 of machines paid for and unused, at $2.50 an hour.

The static split was fair and it was wasteful at once. It wasted hours (idle machines) and it wasted the better experiment (the sweep that did not run). A shared queue does not fix this, because the teams do not agree on whose work matters more, and a priority number set by whoever writes the job is not an answer anyone trusts.

What buckette does. Put the pool on an auction. At each round, on a schedule you choose, buckette allocates the hours among standing bids, the highest price per hour first. Each winner receives its own child bucket split off the pool, and a key that can spend that child alone:

PUT /buckets/<gpu-pool>/auction
{"kind": "quantity",
 "unit": {"gpu_hours": 1},
 "priceUnit": "credits",
 "funding": "funded",
 "schedule": {"tag": "IntervalSecs",
              "contents": 3600},
 "pricing": "uniform",
 "award": {"expiresAfterSeconds": 86400,
           "keyScopes": ["ScopeConsume"]}}

POST /buckets/<gpu-pool>/auction/bids
{"units": 60, "minUnits": 20, "price": 4,
 "payerBucketId": "<team-a-credits>"}

With "funding": "funded", each team bids from a budget of credits of its own: the bid’s whole worth is held there while it stands, and a round spends what the team won. A team that bids high on Tuesday has less to bid with on Thursday, so a price is a real trade-off, not a number anyone can inflate. Team A’s sweep wins the hours C and D did not bid for, and the machines run.

Three things keep it honest:

  • No value is invented. buckette sorts the prices the bidders write and clears the round; it computes no value of its own.
  • Unused awards come back. A team that finishes early reclaims its award, and the rest goes back to the pool for the next round.
  • The auction can stay above a floor. Name a consume class on the auction and the lot stays above that class’s floors, so a share reserved for, say, production inference is never put up for sale.

Priority, value and price

The three stories use three different signals, and they are easy to conflate:

  • Priority orders a queue. It has no unit, and it says “this first”, not “this is worth more”. It is the right tool when you already know the order, as in the release review.
  • Value says what a piece of work is worth, in units you name. buckette records it and does not act on it, so it can be audited later without changing what is served today.
  • Price decides who gets a scarce resource when several parties compete for it and nobody is in a position to set the order for them.

Putting value into priority loses the units, the absence and the record; putting priority into price turns a queue into a market nobody asked for. buckette keeps them apart on purpose.

You cannot price what you did not record

The first post’s costs arrived as invoices. These do not arrive at all unless something writes them down: a refusal that leaves no trace, a queue whose waits nobody measures, a share of the pool that expired unspent. That is why buckette records refused demand per budget and per reason (for consumes, holds, claims and transfers), ranks the budgets that turn the most away, warns before a reset throws part of a balance away, and keeps the value you attach to each claim on every event it writes.

Where to start

Start recording value now, even if nothing reads it yet. Add clrValue to the claims you already make, in whatever unit your team argues in (revenue, customers, hours saved), and send work whose order matters as claims with a priority. A month later you will be able to say what your budgets turned away and what it was worth, which is the conversation the five stages end on.

The getting started page sets up a first budget in a few minutes, and the guide for agents has every route mentioned here.

If your budgets turn away work that mattered, write to hello@buckette.co. We would like to hear what it was worth.