A database incident runbook is a decision framework that helps a small team identify customer impact, stabilize the database, choose containment or failover, validate recovery, and record follow-up work. It should be short enough to use during an outage and specific enough to prevent risky improvisation.
A database incident runbook should define one incident commander, three practical severity levels, and explicit approval for destructive changes or failover. The first objective is not perfect diagnosis. It is to stop the failure from widening while preserving data integrity and a trustworthy recovery path.
Raff Technologies has supported more than 15,000 virtual machine deployments. From Raff’s infrastructure perspective, the most useful operating rule is simple: stabilize user-facing service before optimizing the system, and preserve evidence before making irreversible changes. This guide covers connection exhaustion, disk pressure, replication or primary failure, recovery validation, communication, and post-incident ownership. It belongs to the Database Reliability Operations for Small Teams cluster.
Database incidents require severity and named ownership
A database alert becomes an incident when it threatens customer operations, data correctness, recovery objectives, or the team’s ability to control the system.
Define severity from impact rather than from one infrastructure metric.
| Severity | Practical definition | Response expectation |
|---|---|---|
| SEV-1 | Critical writes fail, data integrity is uncertain, or the primary database is unavailable | Immediate incident command, containment, recovery decision, and stakeholder updates |
| SEV-2 | Partial degradation, elevated errors, replica problems, or capacity pressure with customer impact | Named owner, active mitigation, regular internal updates |
| SEV-3 | Early warning or contained degradation without material customer impact | Investigate during working hours and create preventive work |
Assign these roles even when one person performs several of them:
- Incident commander: coordinates decisions, timeline, and communication.
- Database responder: diagnoses database and replication state.
- Application responder: confirms user impact, connection pools, retries, and recovery.
- Infrastructure responder: checks compute, storage, networking, and host health.
- Communications owner: updates support, leadership, customers, or partners.
The incident commander should not become the only person running commands. Coordination and execution are different responsibilities.
Consequential actions need explicit authority:
- terminating sessions;
- pausing workers or scheduled jobs;
- disabling writes;
- deleting files;
- changing connection limits;
- promoting a replica;
- restoring from backup;
- redirecting application traffic;
- retiring the previous primary.
NIST SP 800-61 Rev. 3 frames incident response as an organization-wide risk-management capability that includes preparation, detection, response, recovery, and improvement. A database runbook applies that structure to data and service decisions rather than treating the outage as a command checklist.
Triage identifies user impact and the active failure mode
Triage should answer five questions before the team makes a high-risk change:
- Which customer workflows are failing?
- Are reads, writes, or both affected?
- Is data correctness uncertain?
- Is the failure getting worse?
- Does the current system still have a safe recovery path?
Collect a minimum evidence set:
| Evidence area | Questions to answer |
|---|---|
| Application | Error rate, latency, connection-pool waits, failed jobs, affected endpoints |
| Database | Active and waiting sessions, locks, query state, transaction age, engine logs |
| Storage | Free space, growth rate, I/O latency, transaction-log or WAL growth |
| Replication | Replica state, lag, replay position, errors, connectivity |
| Infrastructure | CPU, memory, process health, networking, host events |
| Recovery | Last backup, last restore test, available recovery point, current RPO and RTO |
Preserve timestamps. A useful incident timeline records:
14:02 Alert fired 14:04 Customer write errors confirmed 14:07 Connection pool saturation identified 14:10 Background workers paused 14:14 New connection success recovered 14:22 Root cause isolated to retry storm
Do not begin with a broad restart unless the runbook already defines why restarting is safe. A restart can clear evidence, extend recovery, trigger crash recovery, or turn a degraded system into an unavailable one.
The initial classification should use the dominant constraint:
- Connection exhaustion: clients cannot obtain a database connection.
- Disk pressure: storage headroom or write capability is becoming unsafe.
- Lock or query storm: useful work is blocked by long or expensive operations.
- Replication failure: the standby is stale, stopped, or unavailable.
- Primary failure: the active database cannot safely serve traffic.
- Suspected corruption or compromise: data correctness or trust is uncertain.
The response framework matches symptoms to safe actions
The safest action depends on the failure mode, data risk, and recovery objective.
| Failure mode | Immediate safety objective | Lower-risk containment | Escalation decision |
|---|---|---|---|
| Connection exhaustion | Restore admin access and reduce new demand | Pause noncritical workers, reduce retries, shed expensive traffic | Increase capacity only after identifying memory and workload impact |
| Disk pressure | Preserve write and recovery capability | Stop avoidable growth, pause large jobs, add verified capacity | Fail over only when the secondary is current and the primary can be fenced |
| Lock or query storm | Restore useful concurrency | Stop the offending workload or selected sessions | Roll back application change if the workload will immediately return |
| Replica lag | Protect read correctness and failover readiness | Remove stale replica from reads, reduce source pressure | Rebuild or resynchronize when lag cannot recover safely |
| Primary failure | Establish one authoritative writer | Confirm standby health and application reconnect path | Promote only after freshness, fencing, and approval checks pass |
| Suspected corruption | Prevent propagation and preserve evidence | Restrict writes and isolate affected data path | Restore from a known-good point when active copies are not trustworthy |
Connection exhaustion needs load control before limit changes
Connection exhaustion appears when the application or operational tools cannot obtain a usable database session.
Common causes include:
- connection leaks;
- oversized application pools;
- retry storms;
- too many application replicas opening independent pools;
- long-running or idle transactions;
- blocked sessions accumulating behind locks;
- administrative or reporting tools consuming the remaining capacity;
- a slow database causing requests to hold connections longer.
MySQL documents that Too many connections means available connections are in use and reserves an extra administrative connection for privileged diagnosis. PostgreSQL exposes active sessions and waits through its monitoring views. Both examples support the same runbook principle: preserve an administrative path before normal clients consume every slot.
Containment order:
- Confirm that connection acquisition is the user-visible bottleneck.
- Preserve or establish administrative access.
- Reduce incoming demand by pausing noncritical workers, reports, or retries.
- Identify blocked, idle-in-transaction, or unexpectedly long sessions.
- Terminate only sessions whose business impact and rollback behavior are understood.
- Adjust pool behavior before increasing the server limit.
- Validate that new connections and critical transactions recover.
Do not raise max_connections as the first response. More sessions can increase memory use, context switching, lock contention, and I/O pressure while leaving the original workload problem unchanged.
A connection incident is resolved only when the application pool stabilizes. Database availability alone does not prove that clients can recover.
Disk pressure requires write safety and verified headroom
Disk pressure can block useful work, prevent transaction-log writes, break replication, or force database shutdown behavior.
PostgreSQL documentation warns that a full data disk can prevent useful activity and that a full WAL disk can cause server panic and shutdown. MySQL documents full-disk behavior for table and binary-log writes. The practical lesson is to act before the filesystem reaches zero usable space.
Identify what is growing:
- database tables or indexes;
- write-ahead log, binary log, or transaction logs;
- replication slots or retained logs;
- temporary query files;
- backup files stored on the database host;
- application logs;
- failed maintenance artifacts;
- large migrations, index builds, or table rewrites.
Containment order:
- Confirm the affected filesystem and remaining headroom.
- Stop avoidable growth, including large jobs, imports, reports, or backup duplication.
- Preserve transaction logs and database files; do not delete unknown database-managed files.
- Add or extend verified storage when the architecture supports it.
- Remove only files with known ownership, retention, and recovery consequences.
- Check replication and backup chains after capacity is restored.
- Validate writes, checkpoints, log archiving, and application behavior.
Deleting transaction logs manually can turn a capacity incident into a recovery incident. The runbook should name which directories are database-managed and which files are safe for routine cleanup.
Failover requires freshness, fencing, and application validation
Database failover moves the writer role from an unhealthy primary to a standby or replacement system.
Failover is a data-authority decision, not only an availability action. Before promotion, establish:
- primary failure or isolation is confirmed;
- the standby is healthy enough to become primary;
- replica lag and the expected data-loss window are known;
- the promotion satisfies the approved RPO or has business approval for the exception;
- the old primary can be fenced from accepting writes;
- the application can discover and authenticate to the new primary;
- connection pools and workers will reconnect;
- a backup or recovery path remains available.
PostgreSQL documentation emphasizes that a restarted old primary must be prevented from operating as primary after failover because dual-primary behavior can cause data loss. MySQL replication documentation likewise treats failover as a controlled source-switching process rather than a simple restart.
Use this decision test:
| Question | Promote when | Do not promote when |
|---|---|---|
| Is the primary still writing? | It is fenced or definitively unavailable | Write authority is ambiguous |
| Is the standby current enough? | Lag fits the RPO or exception is approved | Recovery point is unknown |
| Is the standby healthy? | Replication and storage state are understood | Errors or corruption are unresolved |
| Can clients reconnect? | Endpoint, credentials, DNS, and pools are tested | Application path is unknown |
| Can redundancy be rebuilt? | A post-failover recovery plan exists | Promotion would leave an unmanaged single point indefinitely |
After promotion, the system is often in a degraded state with no healthy standby. Restoring redundancy is part of incident recovery, not optional later maintenance.
Recovery validation confirms the service, data, and protection path
Recovery is complete only when the agreed service works and the database returns to a protected state.
Validate four layers:
Database process
- engine accepts connections;
- expected primary or writer role is established;
- logs show no unresolved recovery error;
- storage has safe headroom;
- locks and session counts are stable.
Data correctness
- critical tables, schemas, or collections are present;
- recent expected transactions exist;
- known failed or unwanted transactions are absent when recovery targeted an earlier point;
- application-visible totals are plausible;
- permissions and extensions are available.
Application behavior
- connection pools acquire new sessions;
- critical reads and writes succeed;
- workers and scheduled jobs resume safely;
- authentication, billing, ordering, or other key flows work;
- error rate and latency return to an acceptable range.
Protection state
- backups resume;
- transaction-log archiving succeeds;
- replicas are healthy or rebuilding;
- monitoring and alerting reflect the new topology;
- temporary access and emergency rules are reviewed.
Use Database Restore Testing: RPO, RTO, and Recovery Validation when the incident requires historical recovery. Use the Managed Database Migration Checklist when a failed cutover or endpoint change caused the incident.
Communication and evidence make incidents reusable
Database incidents need a communication cadence that reflects severity.
An internal update should include:
- current customer impact;
- confirmed failure mode;
- containment already applied;
- next decision and owner;
- expected time of the next update;
- known data-integrity or recovery risk.
Avoid promising restoration times before the recovery path is measured. State what is known, what is being tested, and when the next update will arrive.
Preserve an incident record:
| Field | Required evidence |
|---|---|
| Start and end | Detection time, customer-impact window, recovery time |
| Scope | Applications, databases, regions, and customer workflows affected |
| Decisions | Session termination, traffic reduction, failover, restore, rollback |
| Data risk | RPO exception, missing transactions, validation results |
| Recovery | Actual RTO, application checks, backup and replica status |
| Follow-up | Named owner, due date, and verification method |
Post-incident review should produce changes, not only a narrative.
Classify follow-up work into:
- detection and alerting;
- application pool and retry behavior;
- capacity and storage headroom;
- query or schema changes;
- replication and failover controls;
- backup and restore readiness;
- runbook clarity;
- ownership and communication.
A useful review asks why the system failed, why safeguards did not contain it, why detection or response took the observed time, and how the team will verify the correction.
Raff supports managed and self-hosted incident paths
Raff supports two database operating models, and the incident runbook should match the chosen responsibility boundary.
With Raff Managed Databases, the service may own more platform operations such as infrastructure maintenance, backup mechanisms, and availability controls. The application team still owns query behavior, connection pools, schema changes, credentials, data validation, incident communication, and the decision to return traffic. Verify the current engine, failover, backup, monitoring, and support boundaries on the live product page.
With a database on a Raff VM, the team owns the full operating path: host capacity, database configuration, replication, backups, failover, recovery, and on-call response. Use Raff Data Protection as an additional infrastructure recovery layer where appropriate while keeping database-native recovery distinct.
A practical self-hosted response path is:
Application impact detected ↓ Database and infrastructure triage ↓ Contain demand or storage growth ↓ Fail over, restore, or rebuild when approved ↓ Validate application and protection state
The operating decision should account for who can respond during connection saturation, disk pressure, failed replication, and after-hours outages—not only the monthly infrastructure price.