Where the limit belongs

NOT at the incoming API level (too coarse). NOT per-agent (too fine, one agent's usage hides). Right level: the shared model client.

Advertisement

Bucket-per-model

RateLimiter geminiLimiter = RateLimiter.create(60);  // 60 QPS
ModelClient limited = new RateLimitedModelClient(baseClient, geminiLimiter);
specialists.forEach(a -> a.setModelClient(limited));
Advertisement

Backoff over drop

Prefer to slow calls (wait for token) over dropping them. Users would rather wait 200ms than see a failure.