K3s Persistent Storage and Backup Strategy

K3s persistent storage is easy to start and easy to misunderstand.

K3s ships with Rancher's Local Path Provisioner, so a PersistentVolumeClaim can get storage from a node's local disk without installing a separate CSI platform. That is an excellent default for development, single-node clusters, and workloads whose recovery plan can tolerate node-local state.

It does not mean the data is highly available.

Once a K3s workload must survive worker-node loss without a manual restore, storage becomes a separate architecture decision. The team may need replicated block storage such as Longhorn, an external CSI-backed storage system, or to move durable application state into a database or object-storage service outside the cluster.

Backup is another separate decision. A K3s datastore snapshot can restore cluster state, but it does not recreate the bytes stored behind an application PVC. Likewise, three replicas of a volume can survive selected hardware failures, but replication does not provide a clean historical copy after deletion, corruption, or a bad application write.

This guide owns that K3s-specific storage decision: Local Path Provisioner, node-loss behavior, when distributed storage is justified, Longhorn as one option, and how to separate K3s datastore recovery from workload-data recovery. For generic Kubernetes PV, PVC, StorageClass, reclaim-policy, and access-mode concepts, use Kubernetes Persistent Storage: Volumes, Storage Classes, and Backups. For the broader DR model, use Kubernetes Backup and Disaster Recovery Strategy.

K3s storage has three separate questions

Before choosing a storage technology, answer three questions independently:

1. Where does the live data sit?
2. What failures should the live storage survive?
3. What historical copy can be restored after corruption, deletion, or cluster loss?

Those questions map to three different controls.

A PVC can provide persistence without high availability. Replication can provide availability without a historical backup. A backup can provide recovery while still allowing several minutes or hours of downtime.

The production design needs the combination that matches the workload.

K3s includes Local Path Provisioner by default

Current K3s documentation ships Rancher's Local Path Provisioner as the built-in local-storage provider.

This gives K3s a local-path StorageClass and allows PVCs to be dynamically backed by storage on the node selected for the workload.

Conceptually:

Pod
  ↓
PVC
  ↓
local-path PersistentVolume
  ↓
Directory or local storage on Node A

The key word is local.

The PV object is managed through Kubernetes, but the application bytes remain on the selected node's storage. The current Local Path Provisioner uses node affinity so the resulting volume is associated with the node where it was provisioned.

That makes local-path much more convenient than manually creating host directories and wiring every workload to them, but it does not turn local disk into network-attached or replicated storage.

What local-path survives — and what it does not

The easiest way to decide whether local-path is safe enough is to trace real failure scenarios.

Kubernetes documentation makes the same underlying limitation clear for node-local volumes: when the underlying node becomes unhealthy, the local volume can become inaccessible and the workload using it cannot simply run elsewhere with the same data.

This is the most important sentence in the guide:

A K3s PVC backed by local-path can outlive a Pod without being able to outlive the node.

Local-path is a strong fit when node loss is a recovery event

Local storage is not automatically a bad production choice.

It is reasonable when the workload's failure model accepts rebuilding or restoring after node loss.

Good fits include:

• development and CI environments;
• single-node K3s clusters;
• caches that can be rebuilt;
• generated artifacts that have an authoritative copy elsewhere;
• internal tools with a longer recovery window;
• application data with a simple, tested off-node backup;
• workloads pinned deliberately to one storage node.

A small application can be more reliable with one well-understood local volume plus tested backups than with a distributed storage system nobody on the team knows how to repair.

At Raff, we use a simple boundary for self-managed K3s: local-path is appropriate when losing the node is an accepted restore event. Once the requirement becomes “the workload must keep its data available after that node disappears,” the storage architecture needs another layer.

Inspect the StorageClass instead of assuming its behavior

The local-path StorageClass hides useful implementation details, so review it before relying on it for production.

At minimum, check:

kubectl get storageclass
kubectl get storageclass local-path -o yaml
kubectl get pv
kubectl get pvc -A

Review:

• provisioner name;
• volume binding mode;
• reclaim policy;
• whether it is the default StorageClass;
• node affinity on provisioned PVs;
• the configured node paths used for storage.

Current upstream Local Path Provisioner manifests commonly use WaitForFirstConsumer, which lets scheduling context participate before local storage is provisioned. The provisioner also supports configurable host paths and node-affinity behavior.

Reclaim policy deserves particular attention. If the StorageClass uses Delete, deleting the PVC can cause the backing volume data to be removed as part of reclamation. If the workload contains business data, deletion protection must come from access controls, backup policy, and restore capability rather than hope.

K3s local-path is persistence, not shared storage

A common mistake is to treat a local-path PVC as if any worker can mount it later.

That is not the operating model.

Node A
  └── local volume A
       ↑
       workload tied to A's storage

Node B
  └── cannot magically mount A's local disk

If the application needs storage that multiple nodes can access or a volume that can be reconstructed on another node, evaluate a different backend.

Possible directions include:

• distributed block storage such as Longhorn;
• an external CSI-integrated block storage system;
• network filesystems for workloads that genuinely require shared filesystem semantics;
• an external database for structured state;
• object storage for uploads, artifacts, media, and other object-shaped data.

The best answer is often to move the state out of the cluster workload rather than make every Kubernetes volume highly available.

For example, a stateless API with external PostgreSQL and object storage is usually simpler to reschedule than an API that stores database files and user uploads on node-local PVCs.

Local-path vs Longhorn vs external state

For small self-managed K3s clusters, these are the three most useful patterns to compare.

There is no reason every workload in one cluster must use the same model.

A production K3s environment may legitimately combine:

local-path
  → caches and low-criticality local state

Longhorn / CSI storage
  → selected stateful cluster workloads

external database
  → relational business data

object storage
  → uploads and durable objects

The storage class should follow the data's failure requirement, not the fact that the application happens to run on Kubernetes.

Longhorn adds replicated block storage to K3s

K3s documentation explicitly supports Longhorn as a distributed block-storage option.

Longhorn uses Kubernetes and CSI integration to create block volumes whose replicas can be distributed across cluster nodes and disks.

A simplified model is:

Application Pod
      ↓
Longhorn PVC
      ↓
Longhorn volume
  ├── replica on Node A
  ├── replica on Node B
  └── replica on Node C

That changes the node-failure model. If the node currently running the workload disappears, a healthy volume can be reattached elsewhere when the required replicas and cluster conditions remain available.

This is the capability local-path does not provide by itself.

Longhorn also adds another storage platform to operate. Nodes need its host prerequisites, including iSCSI tooling for its CSI path, and the team must monitor replica health, disk capacity, rebuild behavior, backup targets, and Longhorn upgrades.

Distributed storage reduces one class of failure by creating a new system that also needs operational ownership.

Replication has a real capacity cost

Longhorn's current default StorageClass uses 3 replicas for replicated volumes.

Each replica stores a full copy of the written volume data. Longhorn's documentation states that with n replicas, x bytes of written user data consumes approximately n × x bytes of physical storage.

So, as a rough planning example:

100 GiB of fully written application data
× 3 replicas
≈ 300 GiB of replica data

That is before allowing for snapshots, rebuild headroom, filesystem behavior, and operational free space.

Longhorn uses thin provisioning, so a 100 GiB declared volume does not necessarily consume 300 GiB immediately. The useful capacity calculation should be based on expected written data plus replication and safety headroom.

This is why distributed storage should not be added casually to a very small K3s cluster. Replication consumes disks, node-to-node network bandwidth, CPU, and recovery capacity during rebuilds.

Longhorn does not make every failure harmless

Replicated storage improves availability, but the application can still fail when:

• too many replica nodes disappear;
• the cluster lacks free disk space for replica rebuilds;
• the network between storage nodes is unstable;
• a bad application write is replicated successfully;
• a user deletes data;
• the PVC or storage objects are removed;
• the whole cluster or storage system is lost;
• backups were never tested.

Replication protects against selected infrastructure failures.

It does not protect against every logical failure.

That is why a three-replica volume still needs a recovery strategy.

Snapshots, replicas, and backups are different

Storage terminology becomes dangerous when all three are treated as synonyms.

If the storage system supports snapshots, use them for fast operational recovery where appropriate.

If the data matters after the cluster or storage platform itself is lost, keep a backup outside that failure domain.

Longhorn supports backup and restore workflows in addition to replicas and snapshots. The exact policy should be designed around the workload's recovery objective rather than using the product defaults as a business requirement.

K3s datastore backup does not back up PVC data

This is the most important backup boundary in self-managed K3s.

K3s datastore backups protect Kubernetes cluster state.

Depending on the topology, K3s may use:

• SQLite;
• embedded etcd;
• an external datastore.

For embedded etcd, K3s provides k3s etcd-snapshot. Current K3s defaults schedule snapshots at 00:00 and 12:00 system time and retain 5 snapshots unless the operator changes the configuration. K3s can also upload etcd snapshots to S3-compatible object storage.

K3s also requires the server token to be preserved for recovery because the token participates in protecting confidential bootstrap data in the datastore.

Those backups can restore objects such as:

• Deployments;
• Services;
• Secrets and ConfigMaps stored in Kubernetes;
• PVC and PV objects;
• node and cluster metadata.

They do not automatically contain the application bytes stored on a local-path directory, Longhorn volume, external block device, database, or object store.

K3s datastore snapshot
  └── knows the PVC exists

Application-data backup
  └── contains the bytes behind the PVC

At Raff, we separate these as two recovery plans. Restoring K3s state can recreate the cluster's view of a PVC; it cannot recreate storage bytes that were never part of the K3s datastore backup.

For embedded-etcd versus external-database backup details, continue with K3s High Availability: Embedded etcd vs External Database.

A K3s backup strategy needs multiple layers

For a stateful production K3s workload, map every important state category to an independent recovery mechanism.

This decomposition prevents a common disaster-recovery failure: successfully restoring Kubernetes while discovering that the business data lived somewhere else and was never protected.

Database files need application-aware backup

A database running inside K3s can use a persistent volume, but the volume is only the storage substrate.

For production databases, volume replication or a filesystem copy is not automatically equivalent to a database-consistent backup.

A proper database recovery plan should consider:

• transaction consistency;
• database-native dump or physical backup tooling;
• write-ahead/transaction logs;
• point-in-time recovery where supported;
• restore order;
• credentials and encryption keys;
• independent restore testing.

This is why moving PostgreSQL or MySQL onto Longhorn does not eliminate database operations.

Use Database Backup Strategy for SaaS Applications for that layer rather than expanding database-specific commands here.

Node loss should drive the storage choice

A simple decision test is to imagine one worker VM disappears permanently at 03:00.

Ask what should happen next.

Outcome A: manual restore is acceptable

If the workload can be down while the team rebuilds the node and restores data, local-path plus off-node backup may be sufficient.

This gives the smallest operating surface.

Outcome B: the workload should restart elsewhere with its data

If the application must resume on another node without first restoring a backup, local-path alone is the wrong availability model.

Evaluate replicated/distributed storage or externalize the state.

Outcome C: the workload should stay stateless

If the application can store durable state in managed databases and object storage, the K3s workload may not need persistent cluster volumes at all.

This is often the simplest scale-out model.

The best production architecture is not the one where Kubernetes owns the most storage. It is the one where each type of state has the clearest failure and recovery owner.

Storage topology should match node topology

Replicated storage can only be as independent as the nodes and disks holding the replicas.

Three replicas are less useful if all three copies depend on the same failure domain.

When operating distributed storage, review:

• which nodes can host replicas;
• whether replicas are separated across independent nodes/disks;
• disk free-space thresholds;
• storage and workload contention;
• rebuild traffic;
• maintenance/drain procedures;
• what happens when a node returns after a long outage;
• whether the cluster has enough spare capacity to rebuild a lost replica.

A storage system that is healthy only while every node is present is not giving the resilience the architecture expects.

For small K3s clusters, reserve recovery headroom rather than filling storage nodes to their practical limit.

Do not make control-plane servers storage nodes by accident

K3s servers can also run workloads, so a small cluster may place application volumes on the same nodes that run the control plane.

That is efficient, but it couples responsibilities:

Server node
  ├── Kubernetes control plane
  ├── datastore
  ├── application Pods
  └── application storage

A disk-pressure or I/O-heavy application can then compete with cluster-state operations.

For low-volume workloads this may be acceptable. For more important clusters, decide deliberately whether server nodes should also participate in application-storage replication.

Role separation becomes useful when:

• storage rebuilds create heavy I/O;
• control-plane latency needs protection;
• storage disks are sized differently from control-plane disks;
• maintenance should isolate control plane from data movement;
• workload and storage nodes need different failure policies.

Do not add dedicated nodes because Kubernetes diagrams look cleaner. Add them when resource isolation or failure isolation justifies the extra infrastructure.

Backups should leave the cluster failure domain

A backup stored only on the same node as the source volume is vulnerable to the same node loss.

A backup stored only inside the same distributed storage system may still be vulnerable to cluster-wide failure, operator mistakes, or storage-system corruption.

For important data, keep at least one recoverable copy outside the primary K3s storage failure domain.

That might mean:

• an S3-compatible backup target;
• another storage system;
• database-native backup storage;
• a separate recovery environment.

K3s itself supports S3-compatible storage for embedded-etcd snapshots. Storage platforms such as Longhorn also provide off-cluster backup workflows.

The tool matters less than the separation: the backup must survive the failure scenario that requires it.

Restore testing is more important than backup success messages

A green backup job proves that a process produced an artifact. It does not prove the application can be recovered.

A useful K3s restore test should answer:

• Can the K3s control plane be rebuilt from the datastore backup?
• Is the original server token available?
• Can the required StorageClass or CSI platform be recreated?
• Can application volumes be restored on replacement nodes?
• Can databases start from a consistent backup?
• Are Secrets and external credentials available?
• Can workloads bind to the restored PVCs?
• Can the application pass real health and data-integrity checks before public traffic returns?

Run restore tests against the recovery procedure, not only against the backup tool.

Restore order for a lost K3s environment

The exact commands depend on the datastore and storage provider, but the dependency order is consistent.

1. Recover K3s control plane
        ↓
2. Re-establish storage provider / StorageClasses
        ↓
3. Restore application volume data
        ↓
4. Restore database state using DB-native procedure
        ↓
5. Reconcile PVCs and workloads
        ↓
6. Validate application data and health
        ↓
7. Restore public traffic

Do not send customer traffic back simply because Kubernetes reports that Pods are Running.

A restored Pod can still have:

• an empty filesystem;
• stale database data;
• missing object data;
• incorrect Secrets;
• a PVC bound to the wrong storage;
• an application that starts successfully but contains incomplete state.

Validate the business data before calling recovery complete.

Reclaim policy is part of deletion risk

StorageClasses and PersistentVolumes have reclaim behavior that controls what happens to the underlying storage when a claim is released.

The generic details belong in the Kubernetes persistent-storage guide, but the production K3s rule is simple:

do not let the default reclaim behavior become your deletion-recovery strategy.

Before production:

1. inspect the StorageClass reclaim policy;
2. know whether deleting a PVC also deletes its underlying data;
3. restrict who can delete PVCs, PVs, and storage resources;
4. keep independent backups for irreplaceable state;
5. rehearse recovery from accidental deletion.

A Retain policy can preserve storage after a claim is released, but it is still not an independent backup. A Delete policy can be operationally convenient, but it makes access control and backup discipline even more important.

How to choose storage for common K3s workloads

This table is a starting point, not a universal rule. Application consistency, latency, access mode, recovery objective, and team expertise can change the decision.

When Longhorn is worth the additional complexity

Longhorn becomes attractive when most of these are true:

• the workload must survive a worker-node failure without a manual data restore;
• the team wants storage controlled from Kubernetes;
• several nodes have reliable disks and private network connectivity;
• replica rebuild traffic is acceptable;
• the cluster has enough free capacity for replicas and recovery;
• the team will monitor and upgrade the storage layer;
• off-cluster backups are part of the design.

Local-path is usually better when most of these are true:

• one node is already an acceptable failure domain;
• the workload is rebuildable;
• restore time can be measured in tens of minutes or hours;
• the team wants the fewest moving parts;
• an external authoritative copy already exists;
• distributed-storage overhead would be disproportionate to the workload.

The correct upgrade trigger is a changed recovery requirement, not the number of Kubernetes nodes.

Production K3s storage checklist

Before calling a stateful K3s workload production-ready, confirm:

• Every PVC has an identified StorageClass and failure model.
• The team knows which node or storage system contains the actual bytes.
• Local-path workloads explicitly tolerate node loss or have a tested restore.
• Replicated volumes have enough independent nodes/disks and rebuild capacity.
• Replica count is included in capacity planning.
• Database backups are application-aware, not only volume copies.
• K3s datastore backup is separate from application-volume backup.
• The K3s server token is stored with the control-plane recovery material.
• Important backups leave the primary cluster/storage failure domain.
• PVC/PV deletion permissions are restricted.
• Reclaim policies are understood.
• Restore order is documented.
• A replacement-node or replacement-cluster restore has been tested.
• Application data is validated before traffic returns after recovery.

Raff VM boundary for self-managed K3s storage

A self-managed K3s cluster on Raff VMs gives your team direct ownership of the node disks, K3s StorageClasses, any CSI or distributed-storage layer, and the backup process.

A simple progression is:

Stage 1
K3s + local-path
  → lowest complexity

Stage 2
K3s + selected external state
  → databases/uploads move outside node-local storage

Stage 3
K3s + replicated CSI storage
  → selected in-cluster volumes tolerate node loss

Keep storage replication and cluster traffic on private networking where possible, especially when a distributed storage layer moves data between worker nodes.

If the team wants Kubernetes workloads without owning the K3s control plane and its surrounding infrastructure lifecycle, compare that model with Raff Kubernetes. The storage decision still exists, but the platform ownership boundary changes.

Final recommendation

For small K3s clusters, start with local-path only when node loss is an acceptable restore event. It is simple, built in, and a strong fit for replaceable or low-criticality state.

When application data must remain available after a worker disappears, evaluate replicated storage such as Longhorn or move the durable state into an external service. Longhorn can add node-failure tolerance, but its replicas consume real disk and network capacity and the storage platform itself must be monitored, upgraded, and backed up.

Most importantly, keep the recovery layers separate. K3s datastore snapshots protect cluster state. PVC/storage backups protect application bytes. Database-native backups protect transactional consistency. Replication protects live availability.

A production design needs the controls required by the workload rather than one technology expected to solve all four problems.

Next, continue with K3s Upgrade and Rollback Strategy, because datastore and application backups are prerequisites for safe cluster-version rollback.

Sources

• K3s — Volumes and Storage
• K3s — Backup and Restore
• K3s — etcd Snapshot CLI
• Rancher — Local Path Provisioner
• Kubernetes — Volumes
• Kubernetes — Persistent Volumes
• Longhorn — CSI on K3s
• Longhorn — StorageClass Parameters
• Documentation verified: 2026-08-24