Per-Tenant API Key Isolation: What Went Wrong With Shared Keys

When a standalone service gets folded back into a shared hub, or more generally whenever two previously separate code paths start sharing infrastructure, per-tenant secret isolation is one of the highest-risk places for a silent bug, because key handling logic that was safe in a single-tenant context can carry an unstated single-tenant assumption straight into a multi-tenant one. In a real consolidation, exactly this happened: a per-tenant API key isolation fix was needed as part of folding a standalone service back into the main hub. The specifics of the fix matched a pattern worth checking in any multi-tenant system handling per-tenant secrets: two independent layers of protection are needed, encryption of the secret at rest, and database-level tenant scoping (row-level security or equivalent) on the table holding it. Either layer alone leaves a path to a breach, encryption without tenant scoping means a compromised service-role query can still read another tenant's decrypted key, and tenant scoping without encryption means a raw database dump exposes everything in plaintext. If you're auditing your own system: confirm both layers exist independently, trace every code path that could read a secrets table with elevated (service-role or admin) privileges and verify it still respects tenant boundaries, and check that secrets never end up logged, echoed in error messages, or included in exported diagnostics, a surprisingly common leak path that bypasses both of the layers above entirely.