Building Scalable APIs: Lessons from Production

Scale is mostly about saying no gracefully: no to unbounded work, no to unlimited connections, no to requests that should have failed fast. These patterns held up serving millions of requests.

Rate limit early

Put limits at the edge before requests reach your logic. Per-key quotas protect you from one noisy consumer, and global concurrency caps protect you from all of them at once.

Pool the database

Never open a fresh connection per request in a hot path. A modest pool with timeouts and prepared statements beats an unbounded one every time, and it fails loudly instead of melting quietly.

1const pool = new Pool({
2  max: 20,
3  idleTimeoutMillis: 30000,
4  connectionTimeoutMillis: 5000,
5  statement_timeout: 10000,
6});

Anti-patterns to retire

• •N plus 1 queries hidden behind innocent-looking loops.
• •Unbounded pagination with ever-growing offsets.
• •Synchronous calls to slow third parties inside request handlers.
• •Retrying without backoff, which turns blips into outages.

Cache at the edge

Not every response needs a fresh computation. Cacheable GETs belong behind short TTLs with explicit invalidation on writes, and per-user data gets keyed cache entries rather than bypassing the cache entirely.

Shed load gracefully

When the system saturates, fail fast and loud instead of queueing forever. Timeouts on every outbound call, circuit breakers on flaky dependencies, and 503 with Retry-After keep one slow dependency from taking the whole API down.

Scale is a property of the design, not the hardware. The same patterns apply at a hundred requests a second and a hundred thousand, which is exactly why they are worth learning once.