A noisy tenant made everyone wait 9.6 seconds: four fixes

In a shared FIFO queue, one tenant's burst of 2,000 AI jobs made every other tenant wait 9.6 seconds at the 95th percentile. A fair queue with no hard cap cut that to 27 milliseconds, and the noisy tenant still finished in 12 seconds. A hard per-tenant concurrency cap gave better numbers for everyone else, and left 12 of 20 workers idle while the noisy tenant took 50 seconds (our lab, 2026).

This is the second post in the series on designing AI and SaaS products. It builds a small worker pool with real timers, floods it with one noisy tenant, and compares four scheduling strategies. It ends with 25 lines of fair-queue code you can paste, plus the per-tenant limits to expose to customers.

What is a noisy neighbor, and why is it worse in AI products?

A noisy neighbor is a tenant whose heavy usage slows down every other tenant on shared infrastructure. A multi-tenant system is one where many customers, called tenants, share the same servers, queues and databases.

AI products make the problem sharper for two reasons. First, each job is slow. A language model call takes seconds, not milliseconds, so a worker is occupied far longer than a typical web request holds a thread.

Second, the most constrained resource is often not yours. Anthropic's rate limit documentation says limits are set at the organization level. Every tenant of your SaaS draws from one shared pool of requests per minute and tokens per minute.

On the Build tier, for example, Claude Sonnet 5.5 allows 5,000 requests, 5 million input tokens and 1 million output tokens per minute. Usage is also lopsided: a Forbes Technology Council column gives the example of 50 power users in a 1,000-seat company generating 80 percent of the inference cost. One tenant that runs a batch import can spend the whole allowance. Every other customer then receives a 429 error, whatever their own usage.

What did the lab measure?

The lab simulates a worker pool with 20 workers. Each job takes 100 milliseconds, a stand-in for a model call. Twenty normal tenants each send two jobs per second. At two seconds into the run, one noisy tenant submits 2,000 jobs at once.

Setting	Value
Workers	20
Job duration	100 ms, real timers
Normal tenants	20, two jobs per second each
Noisy tenant	2,000 jobs at t = 2 s
Pool capacity	200 jobs per second
Burst work	10 seconds of full capacity
Runs	3 per strategy, medians reported
Metric	Time from enqueue to start, for normal tenants

Normal tenants load the pool to about 20 percent on their own. The burst alone needs 10 seconds of the whole pool. The three runs of each strategy agreed within 8 percent on the 95th percentile, so the medians below are stable.

The numbers come from a single Node.js 22 process on an Apple M2 with no network. A real deployment adds queue latency and model calls of several seconds, so absolute waits will be larger. The shape of the results holds.

Why does a shared FIFO queue fail?

A first-in, first-out queue serves jobs in arrival order. When 2,000 jobs arrive at once, every job behind them waits for all 2,000 to finish.

Normal tenants waited 5,439 milliseconds at the median, 9,614 at the 95th percentile and 10,064 at the maximum. The pool ran flat out for 10 seconds, and nothing in the queue knew which tenant owned which job.

This is the default in most home-grown systems, such as one Redis list or one job table polled in order. It works until the first customer runs an import.

Which four fixes did we test?

Four fixes replace the single queue: a cap, a fair queue with the cap, a fair queue without it, and a queue limit. Each one changes how a free worker picks the next job.

A concurrency cap limits how many jobs one tenant can run at the same time. Here the cap was 4. Jobs from a tenant at its cap are skipped, and the next eligible job runs.

Fair queueing keeps one queue per tenant and serves the tenants in rotation, one job each, round robin. A tenant with 2,000 queued jobs gets the same turn as a tenant with one.

Admission control refuses new work when a tenant's queue is full. The refused request gets an HTTP 429 error and a retry hint, rather than waiting.

Strategy	Normal p50	Normal p95	Normal p99	Noisy tenant, time after burst	Jobs rejected
Shared FIFO	5,439 ms	9,614 ms	9,964 ms	10.1 s	0
FIFO with cap of 4 per tenant	0.045 ms	0.113 ms	0.193 ms	50.4 s	0
Fair queue with cap of 4	0.009 ms	0.021 ms	0.036 ms	50.4 s	0
Fair queue, no cap	4.5 ms	27.1 ms	34.2 ms	12.2 s	0
Fair queue, queue limit 100	0.008 ms	0.016 ms	0.037 ms	2.6 s	1,896

Every fix beats FIFO by at least 350 times on the neighbours' 95th percentile (our lab, 2026).

What does a hard cap cost?

A hard cap costs capacity. With a limit of 4, the noisy tenant could use only 4 of 20 workers, so 2,000 jobs took 50.4 seconds instead of 10.1.

Meanwhile the pool was mostly empty. The 20 normal tenants kept about 4 workers busy, the noisy tenant kept 4, and 12 sat idle. A scheduler that leaves workers idle while work waits is not work-conserving.

That trade is right when the cap protects a scarce downstream resource. If each worker also draws from your shared provider rate limit, a cap of 4 per tenant is a sensible fairness rule. If the pool itself is the bottleneck, the cap just wastes it.

Fair queueing without a cap is work-conserving. The noisy tenant used all 20 workers whenever nobody else needed them. Normal jobs still got through, because each time a worker freed up, the next turn went to a tenant with fewer jobs queued.

The price was small. Neighbours waited 4.5 milliseconds at the median, roughly one-twentieth of a job duration, since a worker had to finish its current job first. The noisy tenant took 12.2 seconds instead of 10.1, because normal jobs took about a fifth of the pool.

Fair queueing with no cap costs the noisy tenant 2.1 seconds, while the hard cap costs it 40.3 seconds (our lab, 2026).

When should you reject work instead of queueing it?

Reject work when waiting is worse for the customer than failing fast. A queue limit of 100 jobs per tenant kept neighbours at 0.016 milliseconds at the 95th percentile and finished the noisy tenant's accepted work in 2.6 seconds.

It also rejected 1,896 of its 2,000 jobs, about 95 percent. For a bulk import, that is a bug unless the client retries politely. For an interactive chat, a fast 429 is better than a spinner.

The limit exists to bound memory and wait time. An unbounded queue turns a traffic spike into a delay that grows without limit, and the delay is the real outage. The earlier post on rate limiter algorithms compares the token bucket and sliding-window designs that sit in front of a queue like this.

Return the right signal. A 429 response with a Retry-After header tells well-behaved clients when to come back. Anthropic's own API does this: it returns a 429 and, when the header is present, tells you to wait that many seconds before retrying, per the rate limit documentation.

What does a fair queue look like in code?

A fair queue needs one array per tenant and a rotating pointer. This version is the lab's work-conserving strategy, in plain JavaScript.

class FairQueue {
  constructor() { this.queues = new Map(); this.order = []; this.next = 0; }

  push(tenant, job) {
    if (!this.queues.has(tenant)) { this.queues.set(tenant, []); this.order.push(tenant); }
    this.queues.get(tenant).push(job);
  }

  // Returns the next job, rotating across tenants that have work.
  take() {
    for (let k = 0; k < this.order.length; k++) {
      const i = (this.next + k) % this.order.length;
      const q = this.queues.get(this.order[i]);
      if (q.length) { this.next = (i + 1) % this.order.length; return q.shift(); }
    }
    return undefined;
  }
}

Call take() whenever a worker frees up and whenever a job arrives. To add the concurrency cap from the earlier strategies, skip a tenant whose in-flight count is at the limit. To add admission control, return false from push when a tenant's array has 100 jobs.

This scans all tenants on every call, which is fine for hundreds of tenants. For tens of thousands, keep a list of tenants that currently have work, so each call touches only those.

Which per-tenant limits should you expose?

Each tenant needs limits on three things: requests per minute, tokens per day or month, and concurrent jobs. Requests protect the queue, tokens protect your bill, and concurrency protects the pool. The cost post earlier in this series shows why tokens matter: a single heavy seat can cost more than it pays.

Consider the layers in order:

1. Per-tenant budget. Count tokens per tenant and stop or downgrade at a threshold. This protects your margin.
2. Per-tenant rate limit. Use a token bucket in front of the queue. This smooths bursts before they reach the pool.
3. Fair scheduling. Use per-tenant queues so that whatever gets through is served in turn.
4. Provider limits per workspace. Anthropic lets you set lower rate limits for each workspace, and its documentation gives the example of capping one workspace at 30,000 of 40,000 input tokens per minute to protect the others. Map each large tenant, or each tier of tenant, to its own workspace and key.

Organization-wide limits always apply on the provider's side, even when workspace limits add up to more, according to the same documentation. Your own per-tenant limits therefore have to sum to less than the provider's ceiling, or the provider will make the choice for you.

How should tenants share data?

Scheduling is only half of isolation. Data isolation decides who can read whose rows, and the three common models trade cost against separation.

Model	How it works	Isolation	Cost to run
Pooled	One database, a tenant_id column on every table	Enforced by queries and policies	Lowest
Bridge	One schema per tenant in one database	Stronger, easier per-tenant backup	Medium
Silo	One database or deployment per tenant	Strongest, hardest to operate	Highest

PostgreSQL supports the pooled model with row-level security, a feature that restricts which rows a query can see based on policies defined on the table. The PostgreSQL documentation on row security policies describes how policies attach to tables and apply to every query.

Most AI products start pooled and move large customers to a silo when contracts demand it. Design for that: carry a tenant ID through every queue message, log line and cache key from the first day, so a later split is a migration and not a rewrite.

What should you do first?

Start with the change that costs least and measure again.

1. Tag every job with a tenant ID. Without it, nothing else works.
2. Replace the single queue with per-tenant queues and round robin. It is 25 lines and removes the worst case.
3. Add a per-tenant concurrency cap only if a downstream limit needs it. Otherwise leave the pool work-conserving.
4. Set a queue limit per tenant and return 429 with Retry-After. Choose the number from your longest acceptable wait.
5. Alert on p95 wait per tenant. One tenant's spike should page you before another tenant's complaint does.