A database incident runbook is a decision framework for detecting impact, stabilizing the system, choosing containment or recovery, validating service, and recording what must change afterward.
For a small team, the runbook should be short enough to use during an outage and specific enough to prevent risky improvisation. The first objective is not perfect root-cause analysis. It is to protect data, stop customer impact from widening, preserve a trustworthy recovery path, and make the next decision explicit.
Raff Technologies supports both managed and self-hosted database operating models. Managed Databases can move more infrastructure, backup-platform, monitoring, and availability operations into the service boundary. A database on a Raff VM gives deeper host-level control, but the team owns the complete incident path.
A practical incident sequence is:
Detect impact
-> establish ownership
-> preserve evidence
-> classify failure mode
-> contain safely
-> fail over / roll back / restore when justified
-> validate application + data + protection
-> rebuild redundancy
-> record corrective work
Database incident response starts with impact and ownership
A database alert becomes an incident when it threatens customer workflows, data correctness, recovery objectives, or the team's ability to control the system safely.
Severity should follow impact rather than one infrastructure metric. A lightweight scheme might look like:
| Severity | Practical definition | Response |
|---|
| SEV-1 | Critical writes fail, primary is unavailable, or data integrity may be at risk | Immediate incident command and recovery decision |
| SEV-2 | Partial customer degradation, replica failure, or urgent capacity pressure | Named owner and active mitigation |
| SEV-3 | Contained degradation or warning without material customer impact | Investigate and create preventive work |
The exact labels are a team convention, not a universal standard. What matters is that severity determines ownership, communication, and decision authority.
Assign these responsibilities even when one person performs several roles:
- Incident commander — coordinates decisions, timeline, and communication.
- Database responder — evaluates engine, sessions, locks, replication, and recovery state.
- Application responder — confirms customer impact, pools, retries, workers, and validation.
- Infrastructure responder — checks host, storage, compute, and networking where relevant.
- Communications owner — keeps internal and external stakeholders aligned.
The incident commander should not become the only person executing commands. Coordination and execution are different jobs.
NIST SP 800-61 Rev. 3 treats incident response as part of broader cybersecurity risk management and emphasizes preparation, detection, response, recovery, and improvement. A database runbook applies the same principle to service and data decisions without turning the incident into a command checklist.
Start triage with five questions
Before a consequential change, answer:
- Which customer workflows are failing?
- Are reads, writes, or both affected?
- Is data correctness uncertain?
- Is the condition stable, recovering, or getting worse?
- Does the system still have a safe continuity or historical recovery path?
Collect a minimum evidence set:
| Evidence area | Questions |
|---|
| Application | Error rate, latency, pool waits, failed jobs, affected endpoints |
| Database | Active/waiting sessions, locks, long transactions, engine logs |
| Capacity | Storage headroom, growth, CPU, memory, I/O, connections |
| Replication | Replica state, lag, replay/apply position, errors |
| Infrastructure | Host health, process state, networking, recent events |
| Recovery | Newest usable recovery point, backup/PITR continuity, last restore evidence, RPO/RTO |
Preserve timestamps. A simple timeline is often enough:
14:02 alert fired
14:04 customer write errors confirmed
14:07 connection pool saturation identified
14:10 noncritical workers paused
14:14 new connection success recovered
14:22 retry storm isolated
Do not begin with a broad restart unless the runbook already explains why restart is safe. Restarting can remove evidence, trigger crash recovery, prolong downtime, or turn degradation into complete unavailability.
For the monitoring layer that should trigger this runbook, use Database Monitoring for Small Teams.
Classify the active failure mode before choosing the action
The dominant failure mode determines the safest containment path.
| Failure mode | Immediate safety objective | Lower-risk containment | Escalation decision |
|---|
| Connection exhaustion | Preserve admin access and reduce new demand | Pause noncritical workers, reduce retries, correct pool pressure | Add capacity only after understanding resource impact |
| Disk pressure | Preserve write and recovery capability | Stop avoidable growth, pause large jobs, add verified capacity | Fail over only if secondary and fencing conditions are safe |
| Lock/query storm | Restore useful concurrency | Stop offending workload or selected sessions | Roll back change if workload will immediately return |
| Replica failure/lag | Protect freshness and failover readiness | Remove stale replica from reads, reduce pressure | Rebuild/resync if it cannot recover safely |
| Primary failure | Establish one authoritative writer | Confirm standby/replacement readiness | Promote only after RPO, fencing, and reconnect checks |
| Corruption or compromise | Stop propagation and preserve evidence | Restrict writes and isolate affected path | Restore from a known-good point when active state is untrusted |
This is the core database incident-response rule: match the action to the failure mode instead of applying a generic restart or scale-up response.
Connection exhaustion needs demand control before limit increases
Connection exhaustion occurs when applications or operators cannot obtain usable database sessions.
Common causes include:
- leaked connections;
- oversized pools;
- retry storms;
- autoscaling that multiplies independent pools;
- long-running or idle transactions;
- blocked sessions accumulating behind locks;
- reporting or administration consuming remaining capacity;
- a slow database causing requests to hold connections longer.
Containment order:
- Confirm connection acquisition is the customer-impacting bottleneck.
- Preserve or establish administrative access.
- Reduce new demand by pausing noncritical workers, reports, or retries.
- Identify blocked, idle-in-transaction, leaked, or unexpectedly long sessions.
- Terminate only sessions whose business and rollback effects are understood.
- Correct pool and retry behavior.
- Increase server-side limits only when memory and workload impact are understood.
- Validate that new connections and critical transactions recover.
Do not treat max_connections as the diagnosis. More sessions can increase memory use, context switching, lock pressure, and I/O while leaving the original workload problem unchanged.
A connection incident is resolved only when the application pool stabilizes. Database process availability alone does not prove recovery.
Disk pressure requires write safety and known ownership
Disk pressure can stop writes, disrupt transaction logging, break replication, or make recovery harder.
Identify what is growing before deleting anything:
- database tables and indexes;
- WAL, binary logs, or other transaction logs;
- retained logs caused by replication state;
- temporary query files;
- local backup artifacts;
- application logs;
- failed maintenance output;
- migrations, index builds, or table rewrites.
Containment order:
- Confirm the affected filesystem and remaining headroom.
- Stop avoidable growth such as imports, reports, maintenance, or duplicate backup work.
- Preserve database-managed and transaction-log files.
- Add or extend verified capacity when supported.
- Remove only files with known ownership and known recovery consequences.
- Check replication and recovery chains after capacity is restored.
- Validate writes, logging, backups, and application behavior.
Manual deletion of unknown database-managed files can turn a capacity incident into a recovery incident. The runbook should document what is safe to clean and what must never be removed ad hoc.
Lock or query storms require controlled workload reduction
A database can remain technically available while useful work is blocked by one transaction, migration, or query family.
Confirm:
- which sessions are waiting;
- which session or transaction is blocking them;
- how long the blocker has been active;
- which application or job owns it;
- what rollback would cost;
- whether a deployment or migration caused the change.
A safe response often prioritizes reducing the offending workload over increasing infrastructure.
Possible actions include:
- pause a noncritical worker;
- stop a reporting job;
- disable an expensive feature path;
- reduce retries;
- roll back a recent application change;
- terminate a specific session after understanding rollback effects.
The decision should be reversible whenever possible.
Database failover is a data-authority decision
Database failover moves the writer role from an unhealthy primary to a standby or replacement system. It is not simply a restart or routing change.
Before promotion, establish:
- the current primary is unavailable or can be fenced from writes;
- standby health is understood;
- replica freshness and expected data loss are known;
- the recovery point fits the approved RPO, or an explicit exception is accepted;
- the application can discover and authenticate to the new writer;
- pools and workers can reconnect;
- monitoring and backup paths can follow the new topology;
- there is a plan to rebuild redundancy afterward.
Use this decision test:
| Question | Safer to promote when | Do not promote when |
|---|
| Is the old primary still writing? | It is fenced or definitively unavailable | Write authority is ambiguous |
| Is standby freshness known? | Position/lag satisfies RPO or exception is approved | Recovery point is unknown |
| Is standby health understood? | Storage, replication, and engine state are known | Corruption or unresolved errors exist |
| Can clients reconnect? | Endpoint, credentials, pools, and workers are understood | Application path is uncertain |
| Can redundancy be rebuilt? | A post-failover plan exists | Promotion creates an unmanaged single point indefinitely |
The split-brain risk matters: two systems must not both believe they are the authoritative writer.
After promotion, the incident is not finished merely because traffic works. Rebuilding redundancy, backups, monitoring, and recovery coverage is part of recovery.
For the architectural distinction between failover and historical recovery, use Database Reliability for Small Teams.
Rollback, failover, restore, and rebuild are different decisions
These actions solve different problems:
| Action | Use when | Key risk |
|---|
| Rollback | A recent application/schema change caused the failure and can be reversed safely | Compatibility or partial-change state |
| Failover | Current primary path is unavailable but a trustworthy secondary exists | Data loss or dual-writer state |
| Restore/PITR | Current data state itself is wrong or untrustworthy | Choosing the wrong recovery point |
| Rebuild | Instance/topology is no longer trustworthy or repair is slower/riskier | Longer service recovery and configuration drift |
A team under pressure should not treat these as interchangeable.
When historical recovery is required, use Database Restore Testing: RPO, RTO, and Recovery Validation.
Recovery validation must cover service, data, and protection
Recovery is complete only when customer workflows work and the database has returned to a protected state.
Validate the database
- expected writer role is established;
- connections succeed;
- no unresolved recovery errors remain;
- storage has safe headroom;
- session and lock behavior are stable.
Validate data correctness
- critical schemas/tables are present;
- expected recent transactions exist when they should;
- unwanted transactions are absent when recovery targeted an earlier point;
- roles, permissions, and required extensions are available;
- application-visible totals or invariants are plausible.
Validate the application
- pools create new sessions;
- critical reads and writes succeed;
- workers and scheduled jobs resume safely;
- important business flows work;
- error rate and latency return to an acceptable range.
Validate protection state
- backups resume;
- transaction-log/PITR continuity is healthy where required;
- replicas are healthy or rebuilding;
- monitoring reflects the new topology;
- temporary emergency access is reviewed;
- the team again has a usable recovery path.
A service that is online but has no current backup, replica, or recovery coverage may still be in an incident state.
Communication should state facts, decisions, and uncertainty
An incident update should include:
- current customer impact;
- confirmed failure mode;
- containment already applied;
- current data-integrity or recovery risk;
- next decision and owner;
- what will prove recovery;
- time or trigger for the next update.
Avoid promising a restoration time before the recovery path has been measured. Communicate what is known, what remains uncertain, and which decision is being tested next.
Preserve an incident record:
| Field | Evidence |
|---|
| Timeline | Detection, impact, containment, recovery |
| Scope | Applications, databases, regions, workflows |
| Decisions | Sessions terminated, traffic shed, failover, restore, rollback |
| Data risk | RPO exception, missing data, validation findings |
| Recovery | Actual RTO, application checks, backup/replica state |
| Follow-up | Named owner, due date, verification method |
Post-incident work should remove a failure mode, not only describe it
A useful review asks:
- Why did the failure occur?
- Why did existing controls not prevent or contain it?
- Why did detection take the observed time?
- Why did recovery take the observed time?
- Which decision in the runbook was unclear?
- Which corrective action will be tested, not merely documented?
Classify follow-up work into:
- monitoring and detection;
- application pool and retry behavior;
- capacity and storage headroom;
- query or schema changes;
- replication/failover controls;
- backup and restore readiness;
- access and credentials;
- runbook quality;
- ownership and communication.
Post-incident review is complete when corrective work has an owner and a verification method.
Managed databases change the incident responsibility boundary
With a managed database, more infrastructure operations can sit inside the provider boundary. That may include parts of provisioning, platform maintenance, backup infrastructure, service telemetry, supported recovery workflows, and availability controls depending on the service.
The application team still owns:
- customer-impact detection;
- query and schema behavior;
- connection pools and retries;
- application credentials;
- migration decisions;
- workload-specific RPO/RTO requirements;
- recovery-point selection when applicable;
- application validation;
- traffic-return decisions;
- incident communication.
For Raff, use the live Managed Databases, Managed PostgreSQL, Managed MySQL, and Managed Valkey pages to verify the current service boundary and available capabilities.
Self-hosted databases require a complete incident path
With a database on a Raff VM, the team owns the full response path:
- host capacity;
- database configuration;
- monitoring and alerting;
- connection limits;
- logs;
- storage;
- replication;
- backup/PITR design;
- failover;
- restore;
- patching;
- on-call response.
Raff Data Protection can provide an additional VM-level recovery layer where appropriate, but transactional database recovery should remain distinct when the workload needs engine-aware backups or PITR.
The operating-model comparison should therefore include who can respond safely at 02:00 when connections are exhausted, storage is almost full, or the primary is unavailable, not only infrastructure price.
Database incident runbook checklist
Before relying on the runbook in production, confirm:
- Incident severity is based on customer/data impact.
- An incident commander is named.
- Database, application, infrastructure, and communications ownership are clear.
- Consequential actions have approval rules.
- Administrative database access is preserved during connection exhaustion.
- Safe demand-reduction steps are documented.
- Disk cleanup boundaries are explicit.
- Lock/query containment includes rollback consequences.
- Failover requires freshness, fencing, reconnect, and RPO checks.
- Restore/PITR has a separate decision path from failover.
- Recovery validation includes application and data correctness.
- Backup/replication protection is re-established before incident closure.
- Communication templates state impact and uncertainty clearly.
- Post-incident work has owners and verification criteria.
Conclusion
A database incident runbook gives a small team an ordered way to protect data and restore customer service under pressure.
Start with impact and ownership. Preserve evidence, classify the dominant failure mode, and prefer reversible containment before consequential changes. Connection exhaustion, disk pressure, query storms, failover, and historical recovery require different decisions.
The incident is not complete when the database process starts. Validate customer workflows, data correctness, backup/PITR coverage, replication, monitoring, and the new topology. Then turn the incident into verified corrective work.
Continue with Database Monitoring for Small Teams, Database Reliability for Small Teams, and Database Restore Testing for the detection, architecture, and recovery layers.
