A highly available K3s control plane needs more than multiple server VMs. The cluster datastore must also remain available, consistent, and recoverable when a server fails.
K3s supports two main HA patterns:
- embedded etcd, where three or more K3s server nodes also form the datastore cluster; and
- an external datastore, where two or more K3s server nodes share a separate etcd, MySQL, MariaDB, or PostgreSQL backend.
Both can run production K3s. The better choice depends on what your team wants to operate, which failure domains matter, how you back up cluster state, and whether a reliable external database already exists.
This guide owns that datastore decision. For the broader server/agent, networking, ingress, storage, and operations model, start with K3s on Cloud VMs: Architecture, HA, Storage, and Operations. For K3s versus other Kubernetes operating models, use K3s vs Kubernetes: Which Cluster Model Fits Your Team?.
K3s HA has two different datastore architectures
The control plane stores Kubernetes state such as workload objects, configuration, secrets, node records, leases, and cluster metadata in the datastore.
In HA K3s, that state can live inside the K3s server set or outside it.
Embedded etcd
Stable registration endpoint ↓ ┌────────────┼────────────┐ ↓ ↓ ↓ Server 1 Server 2 Server 3 API + etcd API + etcd API + etcd └────────────┬────────────┘ ↓ Agents
Each server participates in the control plane and embedded etcd cluster.
External datastore
Stable registration endpoint ↓ ┌─────────┴─────────┐ ↓ ↓ Server 1 Server 2 control plane control plane └─────────┬─────────┘ ↓ External datastore PostgreSQL / MySQL / etcd ↓ Agents
The K3s servers share the same external backend rather than maintaining embedded etcd membership themselves.
The important difference is not simply “three servers versus two.” It is where datastore availability and recovery responsibility live.
Embedded etcd is the self-contained K3s HA model
Current K3s documentation defines an embedded-etcd HA cluster as three or more server nodes, with an odd number of servers used to maintain etcd quorum efficiently.
For example:
| Server count | Quorum required | Server failures tolerated before quorum is lost |
|---|---|---|
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
Adding a fourth server to a three-member etcd cluster does not improve failure tolerance. Quorum would become three, so only one server could still fail. This is why odd membership matters.
Embedded etcd is attractive because it keeps the HA dependency set small:
K3s servers = control plane + datastore + etcd membership
There is no separate database service to provision, authenticate, patch, monitor, and recover.
That makes embedded etcd a strong fit when the team wants a self-contained K3s cluster and is comfortable operating quorum, server storage, snapshots, and server-node maintenance.
External databases separate control-plane compute from cluster state
K3s also supports HA with an external datastore. Current K3s documentation lists supported external backends including:
- etcd;
- MySQL;
- MariaDB;
- PostgreSQL.
For this architecture, K3s requires two or more server nodes pointing at the same datastore endpoint.
The architecture separates responsibilities:
K3s server availability ≠ Database availability
Two K3s servers can provide redundant API/control-plane processes, but the cluster is only as available as the external datastore they both depend on.
If both servers connect to one non-redundant PostgreSQL VM, the design still contains a datastore single point of failure.
That means external-database HA is really two availability designs:
- availability of the K3s server layer; and
- availability of the datastore itself.
This model fits best when the organization already knows how to operate the selected database reliably or has a managed database service with suitable availability, backups, monitoring, and recovery.
Embedded etcd vs external database: decision table
| Decision area | Embedded etcd | External database |
|---|---|---|
| Minimum K3s server nodes for HA | 3 | 2 |
| Datastore location | On K3s server nodes | Separate database service |
| Main HA mechanism | etcd quorum | Database HA + redundant K3s servers |
| Extra infrastructure dependency | Lower | Higher |
| Database expertise required | etcd operational basics | Backend-specific DB operations |
| Backup ownership | K3s etcd snapshots | External database backup system |
| Disk sensitivity | High; etcd depends on low-latency storage | Moves datastore I/O to DB tier |
| Scaling control plane | Coupled to etcd membership decisions | Server count less coupled to DB membership |
| Failure isolation | Control plane and datastore share server failure domains | Can separate datastore failure domains |
| Best fit | Small/self-contained HA K3s | Teams with strong database operations or larger cluster needs |
Neither model is inherently more “production” than the other.
The right question is which model produces the smaller and more reliable operating surface for your team.
Quorum is the central embedded-etcd constraint
Embedded etcd uses consensus. A majority of etcd members must be available for the cluster to accept normal state changes.
With three K3s servers:
Server 1 ✓ Server 2 ✓ Server 3 ✕ 2 of 3 available → quorum remains
But:
Server 1 ✓ Server 2 ✕ Server 3 ✕ 1 of 3 available → quorum lost
Existing workloads may continue running for some period when the control plane is impaired, but Kubernetes can no longer reliably reconcile desired state without a functioning datastore and control plane.
This is why embedded-etcd HA should be designed around failure domains rather than simply VM count.
Three VMs that all depend on the same underlying failure domain provide less protection than three servers distributed across independent infrastructure where that option is available.
At minimum, avoid operational actions that can remove quorum at once:
- rebooting multiple server nodes together;
- upgrading all servers simultaneously;
- resizing or replacing several members at once;
- performing storage maintenance across the whole etcd set;
- changing critical network rules without staged validation.
Disk latency matters more with embedded etcd
K3s documentation recommends SSD-backed storage because cluster performance depends heavily on datastore performance. Embedded etcd is particularly sensitive to slow disk I/O.
That is easy to underestimate when the application workloads themselves are lightweight.
A server node has competing responsibilities:
Kubernetes API + scheduler/controller activity + etcd writes + system Pods + optionally application Pods + OS and runtime I/O
If etcd shares a slow or saturated disk with write-heavy application workloads, control-plane health can degrade even though CPU utilization appears acceptable.
For embedded etcd, treat datastore I/O as a control-plane resource rather than leftover disk capacity.
At Raff, our sizing rule is to protect control-plane and datastore headroom separately from application capacity. A server VM should not be considered “large enough” merely because average workload CPU fits on it.
External databases can isolate datastore performance
An external datastore moves cluster-state I/O away from K3s server disks.
That can be useful when:
- the organization already operates PostgreSQL, MySQL, or etcd reliably;
- database monitoring and backups are centralized;
- database storage can be tuned independently from K3s servers;
- larger cluster scale justifies a dedicated datastore tier;
- control-plane servers need to remain simpler or more replaceable.
Current K3s requirements recommend an external database for production and larger clusters, while still supporting embedded etcd as a valid HA model.
The operational trade-off is that K3s health now depends on database network reachability, credentials, TLS, database capacity, and the HA design of that external system.
Moving the datastore outside K3s does not remove database work. It relocates it.
PostgreSQL or MySQL is useful only if the database is actually reliable
A common mistake is to select an external SQL backend because the team already knows SQL, then deploy that database as one ordinary VM.
That architecture looks distributed but is not fully HA:
K3s Server A ─┐ ├── Single PostgreSQL VM ← single failure point K3s Server B ─┘
The database tier needs its own answers for:
- instance failure;
- storage failure;
- backups;
- restore;
- authentication;
- TLS;
- patching;
- monitoring;
- capacity;
- failover;
- connection endpoint stability.
If those controls do not already exist, embedded etcd may actually be the simpler HA architecture.
An external datastore is an advantage when it reuses a mature database operating model, not when it creates a second fragile system beside K3s.
External etcd is different from embedded etcd
K3s can also use an external etcd cluster.
That moves etcd membership outside the K3s server lifecycle:
K3s Server A ─┐ K3s Server B ─┼── External etcd cluster K3s Server C ─┘
This can make sense when an organization already has explicit etcd expertise or needs datastore lifecycle to be independent of K3s servers.
However, the team still owns etcd quorum, certificates, disk performance, backups, upgrades, and recovery. Compared with embedded etcd, external etcd increases architectural separation but not necessarily operational simplicity.
For most small teams, the more useful external-database question is usually whether an existing PostgreSQL or MySQL operational model is stronger than operating embedded etcd inside K3s.
SQLite belongs to the single-server model
K3s uses embedded SQLite by default for a simple single-server installation.
SQLite cannot be used as the datastore for a multi-server K3s cluster.
That gives the datastore progression a clear boundary:
Single K3s server → SQLite is valid HA multi-server K3s → embedded etcd or external datastore
This is why k3s sqlite vs etcd is really a topology decision rather than a generic database benchmark.
SQLite wins on simplicity for one-server clusters. Embedded etcd becomes the built-in HA path once multiple K3s servers must share consistent cluster state.
If a single-server K3s cluster using SQLite needs to become HA, K3s supports conversion to embedded etcd by restarting the server with --cluster-init, after which additional server members can join.
Plan that conversion as a maintenance change: take a current backup, preserve the server token, validate the resulting etcd datastore, and then add HA members deliberately.
Backup behavior differs materially between the two HA models
The datastore choice changes the backup system.
Embedded etcd
K3s provides native etcd snapshot tooling.
Current K3s defaults enable scheduled etcd snapshots twice per day and retain five snapshots unless configuration is changed. Snapshots can also be triggered manually and can optionally be stored in an S3-compatible object store.
A production team should still choose its own retention and off-host policy rather than treating defaults as a recovery objective.
The important point is that embedded-etcd backup is integrated with K3s operations.
External datastore
K3s delegates backup and restore to the external database system.
If the backend is PostgreSQL, MySQL, MariaDB, or external etcd, the database owner must define:
- backup frequency;
- retention;
- off-host copies;
- encryption;
- restore testing;
- point-in-time recovery where supported;
- failover and replica behavior.
K3s does not replace those database-native recovery procedures.
Preserve the K3s server token
K3s documentation also requires backing up the server token at:
/var/lib/rancher/k3s/server/token
The token is used to encrypt confidential bootstrap data stored in the datastore. A datastore backup without the required token can leave the cluster unrecoverable.
This detail matters regardless of how carefully snapshots themselves are stored.
HA and backup solve different failures
A highly available datastore handles selected current failures. A backup protects recoverable historical state.
| Failure | HA datastore helps? | Backup helps? |
|---|---|---|
| One embedded-etcd server fails | Yes, if quorum remains | Usually not needed immediately |
| One K3s server fails with external DB | Yes, if another server + DB remain healthy | Usually not needed immediately |
| Operator deletes resources | No | Yes |
| Bad configuration is replicated | No | Yes |
| Datastore corruption | Possibly not | Yes |
| Whole control-plane environment is lost | Depends on architecture | Yes |
| Credentials/token required for restore are lost | No | Only if token was backed up |
Do not use three etcd members as a substitute for snapshots.
Do not use backups as a substitute for a live HA architecture when the service cannot tolerate control-plane downtime.
External datastore networking becomes part of control-plane reliability
With embedded etcd, datastore traffic stays among K3s server nodes.
With an external database, every K3s server depends on the database path:
K3s servers ↓ private network / secured connection ↓ external datastore
That path should be treated as control-plane infrastructure.
Production design should define:
- private reachability where possible;
- stable DNS or endpoint addressing;
- TLS certificate validation;
- database authentication;
- connection limits;
- latency monitoring;
- firewall rules;
- failure behavior when the database is unreachable.
K3s exposes datastore CA, certificate, and key options for securing external datastore connections where applicable.
Avoid exposing the external datastore publicly simply because multiple VMs must reach it.
Server-count economics should not drive the decision alone
An external database can reduce the K3s server requirement from three HA servers to two, but that does not automatically make it cheaper.
The external datastore itself consumes infrastructure and operational effort.
Compare the complete system:
Embedded etcd 3+ K3s server VMs + snapshots + server maintenance External DB 2+ K3s server VMs + database infrastructure/service + database backups + database monitoring + database lifecycle
If the external database already exists as a trusted shared service, the incremental cost may be low.
If it must be created solely for one small K3s cluster, embedded etcd can be the more efficient architecture.
At Raff, we treat existing operational capability as more important than the raw VM count. Reusing a mature database platform is valuable. Creating a new database platform merely to avoid one K3s server usually is not.
Embedded etcd fits small self-contained HA clusters
Embedded etcd is usually the better starting point when:
- the team wants a self-contained K3s control plane;
- three server VMs are acceptable;
- the cluster is small or moderate in scale;
- the team does not already operate a strong external database platform;
- snapshots can be managed as part of K3s operations;
- server disks are SSD-backed and monitored;
- quorum-aware maintenance is understood.
The biggest advantages are fewer external dependencies and a simpler ownership map.
The biggest risks are quorum mistakes, slow server disks, and coupling datastore lifecycle to server-node operations.
External database fits teams with established database operations
An external datastore is stronger when:
- PostgreSQL, MySQL, MariaDB, or etcd is already operated as a reliable internal service;
- database HA is stronger than what the K3s team wants to build with embedded etcd;
- cluster scale justifies dedicated datastore resources;
- database performance needs to be tuned separately from control-plane VMs;
- K3s server replacement should not alter datastore membership;
- database backup/restore is already standardized and tested.
The biggest advantages are separation of concerns and the ability to reuse database operational maturity.
The biggest risks are a hidden external single point of failure, network dependency, credential/TLS complexity, and a larger total system to troubleshoot.
Failure-mode comparison
| Failure | Embedded etcd | External database |
|---|---|---|
| One K3s server fails | Cluster remains available with quorum | Other K3s server(s) can continue if DB healthy |
| Majority of etcd servers fail | Control plane loses datastore quorum | Not applicable to K3s servers; depends on DB HA |
| External DB endpoint fails | Not applicable | Control plane loses datastore access |
| Slow K3s server disks | Can directly hurt etcd | Less datastore impact, though node health still matters |
| DB network partition | Not applicable beyond server mesh | Can isolate all K3s servers from state |
| Bad cluster-state change | Replicated by etcd | Written to external DB |
| Need historical restore | K3s etcd snapshot | DB-native restore |
| Replace one K3s server | etcd membership must remain healthy | Usually simpler from datastore-membership perspective |
This table is the practical reason datastore choice belongs in its own guide rather than a paragraph inside the broader K3s architecture page.
Migration should be driven by a changed requirement
Do not move from embedded etcd to an external database because the latter appears more “enterprise.”
A datastore migration introduces risk and should solve a named problem such as:
- control-plane scale;
- disk I/O contention;
- centralized database governance;
- standardized DB backup requirements;
- independent datastore lifecycle;
- a need to reuse an existing managed database platform.
Likewise, moving a simple SQLite cluster to embedded etcd makes sense when the requirement changes from one-server simplicity to multi-server HA.
The best datastore is the one whose failure, backup, maintenance, and recovery model the team can operate confidently.
Decision framework
Choose embedded etcd when most of these are true:
- you want the smallest external dependency set;
- three K3s servers are acceptable;
- the cluster is not large enough to justify a dedicated DB tier;
- SSD-backed server storage is available;
- your team can operate etcd quorum and snapshots;
- a self-contained K3s platform is desirable.
Choose an external datastore when most of these are true:
- you already operate a reliable PostgreSQL, MySQL, MariaDB, or etcd platform;
- database backup and failover are mature and tested;
- separating datastore performance from K3s server disks is useful;
- larger cluster requirements justify dedicated DB resources;
- database governance or centralization is an organizational requirement;
- the DB endpoint can remain private, stable, and monitored.
Avoid an external datastore when it would be a single ordinary DB VM with weaker recovery than embedded etcd.
Avoid embedded etcd when the team cannot safely maintain quorum, disk health, or snapshot recovery.
Raff VM architecture for K3s HA
Self-managed K3s can run on Raff VMs using either datastore model.
An embedded-etcd design can use:
3 K3s server VMs ↓ embedded etcd quorum ↓ agent VMs as workload demand grows
An external-database design can use:
2+ K3s server VMs ↓ external HA datastore ↓ agent VMs
Keep K3s server, datastore, and node communication on private networking wherever the environment allows, and size control-plane capacity independently from application workloads.
If your team does not want responsibility for datastore quorum, control-plane lifecycle, upgrades, and cluster recovery, compare the self-managed VM path with Raff Kubernetes instead of adding more self-managed infrastructure by default.