Duplicate usage events overbilled an AI SaaS bill by $115

A plain usage counter overbilled $3,266.75 of AI usage by $114.86, a 3.5 percent error, because the queue delivered 6,984 events twice. All 500 tenants were billed wrongly. Recording each event once under its own ID fixed every tenant and still handled 104,000 events per second on SQLite. The fix is a ledger keyed by event ID, summed at invoice time, and it costs one extra index (our lab, 2026).

This is the third post in the series on designing AI and SaaS products. It simulates 200,000 usage events with duplicate deliveries and crash replays, compares four ways to count them, and measures the error and speed of each. It also covers how to send the totals to Stripe and the rounding mistake that costs more than duplicates do.

What is usage metering, and why is it hard for AI products?

Usage metering is recording how much of a billable resource each customer consumes, so you can invoice the right amount. For an AI product, the resource is usually tokens, and the price is per million tokens.

AI makes metering harder than counting API calls. The cost of a request is unknown until the model responds, because the output length decides half of it. A request can be retried after a timeout. A stream can end early. An agent can make 30 model calls for one user action.

At-least-once delivery is a guarantee that a message will arrive one or more times, never zero. Most queues work this way. If a worker processes a message and crashes before acknowledging it, the queue sends the message again.

A usage event here is one record of the form: tenant, request ID, input tokens, output tokens. If your counter adds every delivery, a redelivery bills the customer twice.

Metering also decides who pays for heavy use. A Forbes Technology Council column gives the example of 50 power users in a 1,000-seat company generating 80 percent of the inference cost. Flat seat pricing hides that. Metered billing exposes it.

What did the lab simulate?

The lab generates 200,000 usage events for 500 tenants, with heavier tenants more likely. Each event carries input and output tokens priced at the Claude Sonnet 5.5 rates on Anthropic's pricing page, which are $2 and $10 per million tokens.

Those prices work out to exactly 2 and 10 micro-dollars per token. Costs are therefore whole numbers of micro-dollars, and the true total is exact: 3,266,748,970 micro-dollars, or $3,266.75.

Fault or setting	Value
Events	200,000
Tenants	500
Duplicate deliveries	3 percent of events, redelivered within 50 messages
Crash replays	2, each resending the last 500 events
Total deliveries	206,984
Database	SQLite via node:sqlite, WAL mode, Node.js 22 on Apple M2
Runs	3 per design and batch size, medians reported

The duplicates model a worker that finishes a job but loses its acknowledgement. The replays model a consumer that restarts and resumes from an older offset. Both are normal behavior for queues such as SQS, Kafka and Redis streams.

Which four ways of counting did we compare?

The first design adds every delivery to a per-tenant counter. The second records each event ID in a table, and adds to the counter only when the ID is new. The third stores every event in a ledger with the event ID as the primary key, and sums at invoice time. The fourth, covered below, rounds each event before summing.

An idempotency key is a unique identifier sent with a request so that repeating the request has no extra effect. For metering, the event ID is the key.

A ledger is an append-only table of events, never updated in place. Totals are computed from it, so you can always explain a number by listing its rows.

Design	Billed total	Error	Tenants wrong	Events per second, batch of 1	Events per second, batch of 100
Naive counter	$3,381.61	+$114.86	500 of 500	77,900	948,000
Counter with seen-table	$3,266.75	$0.00	0	53,400	496,000
Ledger keyed by event ID	$3,266.75	$0.00	0	26,900	104,000

Both deduplicating designs matched the true total exactly, and the naive counter overbilled every tenant (our lab, 2026).

Why does the naive counter fail?

The counter adds whatever it receives. It has no memory of what it has already counted, so a redelivered event is indistinguishable from a new one.

In the lab, 6,984 extra deliveries added $114.86. That is 3.5 percent of the bill, and it was spread across all 500 tenants. No single customer would notice. Across a year, a few percent of revenue is billed incorrectly, and the first customer who audits their invoice finds it.

A counter also cannot explain itself. When a customer disputes a number, there are no rows to list.

How fast is a deduplicating ledger?

A ledger with a unique event ID ingested 104,000 events per second at a batch size of 100. A bare counter ran at 948,000, so deduplication costs roughly nine times the throughput. The ledger is still far above what most AI products produce.

At the lab's 200,000 events per day, the average rate is 2.3 events per second. Even autocommit mode, one transaction per event, ran at 26,900 events per second for the ledger. The batch size mattered more than the design: batching sped up the ledger 3.9 times, and the counter 12 times.

Batching writes in groups of 100 gave every design a speedup between 3.9 and 12 times (our lab, 2026).

The ledger is also compact. Its 200,000 rows took 9.1 megabytes on disk, about 45 bytes per event. Summing every tenant's total at invoice time took about 130 milliseconds, from scratch.

The write timings at batch size 1 varied a lot between runs. The ledger took between 5.9 and 23.2 seconds for the same 206,984 deliveries, because every commit waits on the disk. Treat them as a ceiling, not a benchmark.

What is the safest design in practice?

Use the ledger as the source of truth and add a counter only if you need fast reads. A counter is a cache of the ledger, not a replacement.

The counter-with-seen-table design is exact only when the two writes happen in the same transaction. If the process crashes between "record the ID" and "add to the counter", the event is marked seen and never counted. That is a silent undercount, and it is the hardest bug of the four to find.

Keep both and reconcile them. A nightly job that compares each tenant's counter with the ledger sum catches drift in minutes. In the lab the reconciliation of 500 tenants is one GROUP BY that took about 130 milliseconds.

Reconcile against the provider too. Once a month, compare the token total in your ledger with the usage page of your model provider. A ledger that is lower than the provider's invoice means some calls were never metered, and those are costs you paid without billing anyone.

Here is the ledger in SQL. It runs on SQLite, and PostgreSQL accepts the same ON CONFLICT DO NOTHING form.

CREATE TABLE ledger (
  id      TEXT PRIMARY KEY,   -- the event's idempotency key
  tenant  TEXT NOT NULL,
  hour    INTEGER NOT NULL,
  micro   INTEGER NOT NULL    -- micro-dollars, integer
);

-- ingest: safe to call twice with the same id
INSERT INTO ledger (id, tenant, hour, micro)
VALUES (?, ?, ?, ?)
ON CONFLICT (id) DO NOTHING;

-- invoice
SELECT tenant, SUM(micro) AS micro FROM ledger GROUP BY tenant;

The ON CONFLICT clause is SQLite's upsert syntax. Build the ID from the logical request, such as your own request ID, not from the delivery or the model attempt.

Which rounding mistake costs more than duplicates?

Rounding each event before summing is the bigger risk. Rounding every event to the nearest cent billed $3,321.07, which is $54.32 too much. Rounding every event up to the next cent billed $4,209.47, which is $942.72 or 28.9 percent too much.

The reason is size. A typical event here costs about 1.6 cents, so a whole cent of rounding is a large fraction of it. Small AI requests cost fractions of a cent, and per-event rounding turns them into whole cents.

Floating-point drift, by contrast, did not matter. Summing 200,000 costs as JavaScript numbers in dollars differed from the exact total by less than a ten-billionth of a dollar. The lesson is to use integers anyway, because the fix is free.

The rule that follows is simple. Store cost as an integer in the smallest unit you can, such as micro-dollars. Add the integers. Round once, at the invoice line.

How do you send usage to Stripe?

Send each event with a stable identifier, or send closed hourly totals. Stripe's documentation on recording usage says every meter event has an identifier you can specify, and it recommends idempotency keys to prevent reporting an event more than once.

Four facts from that page shape the design:

1. Timestamps. An event's timestamp must be within the past 35 calendar days and no more than 5 minutes in the future.
2. Finalization. Stripe might accept an event after an invoice finalizes without adding it to that invoice or the next one. Report usage before the invoice closes.
3. Rate. The live-mode meter event limit is 1,000 calls per second per account, with one concurrent call per customer per meter.
4. Pre-aggregation. Stripe suggests combining usage into fewer events if you expect to exceed the limits, and offers a higher-throughput API for up to 10,000 events per second.

Pre-aggregating pays off at once. The 200,000 events fall into 12,000 tenant-and-hour buckets, which is 16.7 times fewer calls. Send a bucket only after the hour closes, with the identifier built from tenant and hour. Meter events add up, so re-sending a bucket that later changed would count it twice.

Your ledger remains the record. Stripe holds the invoice, and the ledger explains it. Keep the rows for as long as customers can dispute a bill.

What goes wrong with AI-specific events?

Three cases need a decision before you write code.

1. Retried model calls. If you retry a failed request, the provider may bill you for each attempt. Decide whether the customer pays for attempts or only for the answer, and key events by logical request so a retry cannot bill twice.
2. Interrupted streams. A user closes the tab halfway through a response. Take the token counts from the provider's reported usage, not from the text you received.
3. Agent loops. One user action can trigger many calls. Meter each call as its own event with a shared parent ID, so a customer can see why one action cost $0.40.

None of these were in the lab, and each deserves its own test in your system. They are the cases where an idempotency key built from the wrong thing double-counts.

What should you do first?

Start with the ledger, because everything else can be rebuilt from it.

1. Create the ledger table with a unique event ID. One migration, and every later step depends on it.
2. Generate the event ID where the work happens. Put it in the job before the first attempt, so every retry reuses it.
3. Store costs as integers. Micro-dollars work for token pricing, and you round once at the invoice.
4. Batch inserts in groups of about 100. It is the cheapest speedup in the lab.
5. Reconcile nightly. Compare ledger sums with whatever you send to Stripe, and alert on any difference.
6. Aggregate before you send to Stripe. Use closed hourly buckets with stable identifiers.