A 24‑Hour Containment Runbook for Exposed API Keys and Webhooks

When an API key or webhook is exposed, attackers can pivot quickly—sending fraudulent requests, scraping data, or triggering unintended actions. This 24‑hour, beginner‑friendly runbook guides you through confirming the exposure, cutting off misuse with minimal downtime, rotating secrets safely, and hardening your environment so the same issue doesn’t happen again.

What “Exposed” Means and Why Minutes Matter

API keys and webhook URLs act like passwords for systems to talk to each other. Exposure can happen through a public Git repo, a screenshot, a crash log, a misconfigured CI job, or a leaked config file. If an attacker sees a valid key or webhook URL, they may:

• Make authenticated API calls to exfiltrate data or modify settings.
• Flood your endpoints with fake webhook events to trigger workflows.
• Abuse metered services, accruing costs or hitting rate limits.
• Plant persistence (new keys, OAuth apps, forwarding rules) for later access.

Assume an exposed key or webhook has been or will be used. The goal is fast containment first, then careful recovery.

Before You Start: Gather the Essentials

Identify what you’re dealing with and where you can act:

• Provider consoles: Cloud, SaaS platforms, payment gateways, email/APIs, messaging, automation tools.
• Code and config sources: Git repos, CI/CD systems, secrets stores, container/task definitions, environment variables.
• Observability: API logs, webhook delivery logs, WAF/CDN logs, SIEM, billing/usage dashboards.
• Contacts: On‑call teammates, managed service providers, and support escalation paths.

0–60 Minutes: Confirm, Contain, and Stop the Bleed

Step 1: Verify What Was Exposed

• Identify the exact secret(s): API key, token, OAuth client secret, signing secret, or webhook URL with embedded token.
• Match the secret to its scope (what systems it can access) and environment (prod vs. non‑prod).
• Check whether the secret is still valid and where it’s configured (apps, CI, serverless functions).

Step 2: Freeze Risk with Minimal Breakage

• Enable provider‑side protections: Tighten IP allowlists, geofencing, and rate limits for the affected key/endpoint.
• Throttle suspicious traffic: Use WAF/CDN to block traffic patterns or known malicious IPs targeting the endpoint.
• Disable only if safe: If you can disable the key or webhook without breaking critical workflows, do it now. Otherwise, proceed to rotation first in a controlled way.

Step 3: Snapshot Evidence

• Export last 7–30 days of API and webhook logs, plus current configuration and usage/billing trends.
• Record timestamps, IPs, user‑agents, request paths, and any object IDs touched for later review.

60–180 Minutes: Rotate and Restore Safely

Step 4: Generate Replacements

• Create new API keys/tokens with the minimum permissions required. Prefer short‑lived tokens or keys that can be limited by IP and scope.
• For webhooks, generate a new signing secret and, if the URL contains a secret token, create a new URL entirely. If possible, move the token from the URL into headers.

Step 5: Stage the Cutover

• Update secrets in a central secrets manager (e.g., cloud KMS, parameter store, vault) rather than hard‑coding.
• Update consuming apps, serverless functions, and CI jobs to use the new secret via environment variables.
• Verify that all dependent services can reach the provider using the new key from a staging or maintenance mode.

Step 6: Switch Traffic and Invalidate the Old

• Rotate one system at a time if the blast radius is large, or perform a coordinated window if you have a small footprint.
• Once all critical paths are on the new key, immediately revoke or delete the old key.
• For webhooks, reject requests signed with the old signing secret and, if applicable, decommission the old URL path.

180–360 Minutes: Hunt for Abuse and Close Gaps

Step 7: Review Logs for Misuse

• Look for traffic from unusual IPs or geographies, spikes in error/success rates, and calls to sensitive endpoints (export, list‑all, delete, permission changes).
• Check whether any new API keys, OAuth apps, forwarding rules, or webhooks were created using the compromised credentials.
• Inspect billing for anomalies: sudden cost increases, unusual service tiers, or new regions used.

Step 8: Lock Down Configuration

• Apply least privilege to all replacement keys (read vs. write, limited resources, IP allowlists).
• Enforce signing verification for webhooks and validate timestamps, replay protection, and event sources.
• Disable anonymous inbound endpoints and remove any hard‑coded tokens from code or build logs.

6–12 Hours: Clean Up the Leak Source

Step 9: Purge Secrets from Everywhere

• Search across repos, wikis, tickets, and logs for the leaked secret and remove or rotate where found.
• If committed to Git, perform a history rewrite (e.g., filter‑repo) to remove the secret, then force‑push and rotate any related deploy keys.
• Invalidate caches, CI artifacts, container layers, and build logs that might still contain the secret.

Step 10: Improve Handling Going Forward

• Introduce pre‑commit and CI scanners for secrets to block future commits with keys or tokens.
• Move environment‑specific secrets to a central store with automatic rotation and role‑based access.
• Document a standard rotation playbook for each provider you use (where to create, rotate, and revoke keys).

12–24 Hours: Communicate, Monitor, and Prevent Recurrence

Step 11: Notify Stakeholders as Needed

• Summarize impact, timeframes, what was exposed, containment steps, and whether any data access is suspected.
• If customer data or financial operations may be affected, prepare customer notices and reach out to the provider’s security team for assistance.
• Update status pages or internal comms channels as appropriate.

Step 12: Increase Monitoring Sensitivity

• Set alerts for: new key creation, permission changes, webhook endpoint changes, unusual request rates, and large exports.
• Enable anomaly detection where supported by your provider.
• For any potential spill of personal or financial data, add credit and identity monitoring to your recovery checklist. If you suspect exposure of customer or billing info, consider using a service like SmartCredit for ongoing privacy, credit monitoring, and identity protection to help watch for suspicious financial activity.

Quick Decision Matrix: Disable vs. Rotate First

Use this to decide your first containment move:

• Disable immediately if: the key is high‑risk (admin, write to payment/user data), you can tolerate brief downtime, or you already see active abuse.
• Rotate first if: the key is critical to uptime (payments, auth), and you can stage a replacement within minutes. Use rate limits and IP allowlists as a temporary shield during rotation.

Webhook‑Specific Hardening Checklist

• Require HMAC verification with a unique signing secret per environment.
• Validate timestamps and replay protection (reject old or duplicate signatures).
• Prefer POST with headers for auth; avoid tokens in query strings or path segments.
• Put webhooks behind a dedicated endpoint with IP allowlists and strict schema validation.
• Log only minimal metadata; never log raw secrets or full payloads containing sensitive data.

Common Pitfalls to Avoid

• Partial rotation: Rotating the key in one service but forgetting a cron job, serverless function, or staging environment.
• Leaving old webhooks active: Attackers replay old signed events if the signing secret isn’t rotated.
• Overbroad scopes: Using a full‑access key where a read‑only or single‑resource key would suffice.
• URLs as secrets: Treating webhook URLs as public; if a token is in the URL, it is a secret and must be rotated.
• Logging leaks: Printing secrets during debugging or error handling; scrub logs and mask sensitive values.

Minimal Evidence to Keep for Post‑Incident Review

• Timeline of exposure discovery and containment actions.
• Before/after configuration snapshots and permission sets.
• Access logs highlighting suspicious activity and IPs.
• List of systems where the secret was present and confirmation of removal.
• Follow‑up tasks with owners and due dates.

Template: 24‑Hour Runbook You Can Copy

1. Confirm scope (0–30m): Identify exposed key/webhook, environment, and privilege level.
2. Contain (0–60m): Tighten rate limits/IP allowlists; disable if safe. Snapshot logs.
3. Replace (60–120m): Generate least‑privilege replacements; update secrets manager.
4. Cutover (120–180m): Deploy updates; validate; revoke old secrets immediately.
5. Hunt abuse (3–6h): Review logs, billing, and config for misuse or persistence.
6. Clean source (6–12h): Remove secrets from code, history, artifacts, and docs.
7. Communicate (12–18h): Notify stakeholders; document impact and next steps.
8. Harden/monitor (18–24h): Enforce least privilege, webhook signing, alerts, and periodic rotations.

Frequently Asked Questions

How do I know if a key was actually used?

Check provider logs for the key ID, client ID, or associated IPs. Look for access outside your normal IP ranges, unusual endpoints, or spikes in 4XX/5XX that indicate probing. Billing anomalies are also a signal.

Do I need to rotate all keys if only one leaked?

Rotate only those in the same privilege family or where you suspect lateral movement. However, if your secret handling process is weak (e.g., keys committed broadly), plan a broader rotation and adopt a central secrets manager.

What if rotation might break production?

Stage the new key in parallel, validate in a canary environment, tighten limits temporarily, then perform a coordinated cutover. Have a fast rollback plan if the new key fails, but keep the old key disabled once misuse risk is high.

Are webhook signing secrets enough?

They’re essential, but also add IP allowlists, replay protection, strict schema checks, and separate secrets per environment. Avoid including tokens in URLs; prefer header‑based authentication.

Conclusion

Exposed API keys and webhooks are time‑sensitive incidents. Act quickly to contain risk, rotate safely, and verify that nothing malicious slipped through. With a clear 24‑hour plan—confirm, contain, replace, cut over, hunt, clean up, communicate, and harden—you can stop abuse fast and restore trust in your integrations. Build prevention into your daily workflow with least privilege, central secret management, webhook signing, and automated secret scanning so the next incident is shorter, cheaper, and less disruptive.