K3s on Cloud VMs: Architecture, HA, Storage, and Operations

K3s is a fully compliant Kubernetes distribution that packages the control plane, container runtime, networking, DNS, ingress, load-balancing support, network policy, and local storage tooling into a smaller operating surface. On cloud VMs, that makes K3s a practical self-managed cluster model for teams that need Kubernetes scheduling without assembling every cluster component from scratch.

The important decision is not whether K3s is “lightweight Kubernetes.” It is which K3s architecture your workload requires and which cluster responsibilities your team is prepared to own.

A single K3s server can run a complete cluster. A production HA design changes the datastore, control-plane topology, network requirements, storage plan, backup process, and maintenance model. This guide owns that architectural overview. Detailed datastore design belongs in K3s High Availability: Embedded etcd vs External Database, while deeper networking, storage, and upgrade procedures belong in their dedicated guides.

K3s architecture has server and agent roles

K3s uses two main node roles.

• Server nodes run the Kubernetes API and control-plane services. They also participate in the cluster datastore according to the selected topology.
• Agent nodes run application workloads and connect back to the servers.

A basic multi-node layout looks like:

                K3s server
        API + control plane + datastore
                    ↓
          ┌─────────┼─────────┐
          ↓         ↓         ↓
       Agent 1   Agent 2   Agent 3
       workloads workloads workloads

The current K3s quick-start documentation confirms that a single server node is already a fully functional Kubernetes cluster: it can run the datastore, control plane, kubelet, and container runtime needed to host workload Pods.

That means K3s supports a useful progression:

Single VM
  ↓
One K3s server running workloads
  ↓
Add agents for workload capacity
  ↓
Add HA servers when control-plane availability justifies it

This is different from treating every production K3s deployment as a three-server cluster from day one.

A small internal application may tolerate one server with a tested recovery process. A customer-facing platform with a short recovery target may justify an HA control plane before workload scale alone would require it.

Single-server K3s is the smallest cluster model

A single-server installation is the lowest-complexity K3s architecture.

Cloud VM
  ├── K3s server
  ├── Kubernetes API
  ├── control-plane components
  ├── default datastore
  ├── containerd
  └── application workloads

The default datastore for a simple single-server K3s cluster is SQLite. K3s also supports embedded etcd and external datastores, but SQLite keeps the smallest topology simple.

A single-server design can fit:

• development environments;
• CI or test clusters;
• internal tools;
• low-criticality production workloads;
• small systems where fast rebuild and restore are acceptable;
• teams learning Kubernetes operations before adopting HA.

The main limitation is clear: the server VM is both a control-plane and infrastructure failure domain.

If that VM fails, Kubernetes cannot schedule new workloads or serve its API until the server is recovered or rebuilt. Workloads on that same VM also disappear with the host.

Adding agent nodes gives more workload capacity, but it does not make a single-server control plane highly available.

Single server
    ↓
Agent 1
Agent 2
Agent 3

If the server becomes unavailable, existing workloads on agents may continue running for a period, but cluster control and normal reconciliation are impaired. That architecture should therefore be chosen deliberately rather than described as HA merely because several VMs exist.

HA K3s changes the datastore and quorum model

K3s supports two primary HA datastore patterns:

For embedded etcd, current K3s documentation requires three or more server nodes and recommends an odd number so etcd can maintain quorum efficiently.

A typical embedded-etcd topology is:

             Fixed registration address
                       ↓
          ┌────────────┼────────────┐
          ↓            ↓            ↓
      Server 1      Server 2      Server 3
      control       control       control
      + etcd        + etcd        + etcd
          └────────────┬────────────┘
                       ↓
                 Agent nodes

Quorum determines whether the datastore can continue accepting changes. With three etcd members, two must remain available for quorum.

Fast, reliable storage matters because etcd is sensitive to disk latency. K3s explicitly warns that embedded etcd can suffer on slower storage.

An external-database HA design looks different:

          Fixed registration address
                    ↓
            ┌───────┴───────┐
            ↓               ↓
        Server 1         Server 2
        control          control
            └───────┬───────┘
                    ↓
            External datastore
          PostgreSQL/MySQL/etc.
                    ↓
                Agent nodes

The external datastore can simplify some control-plane topology decisions, but it creates another production system that must itself be highly available, backed up, monitored, secured, and reachable.

Use the dedicated K3s High Availability guide when choosing between those two models. This connector guide keeps the distinction architectural.

A fixed registration address simplifies HA node joins

In an HA cluster, agents and additional server nodes need a stable endpoint for registration.

K3s supports using a fixed registration address in front of the server nodes. That address can be provided by a TCP load balancer, virtual IP, or another stable endpoint appropriate to the environment.

Agents
  ↓
Stable registration endpoint
  ├── K3s Server 1
  ├── K3s Server 2
  └── K3s Server 3

After registration, K3s agents establish their normal connections to server nodes.

The registration endpoint solves control-plane reachability. It should not be confused with the application ingress path used by customer traffic.

That separation is important:

• control-plane endpoint → Kubernetes API and node registration;
• application ingress → HTTP/HTTPS or other workload traffic;
• ServiceLB or another load-balancer implementation → Kubernetes LoadBalancer Services.

Do not use one vague “load balancer” label for all three responsibilities in the architecture diagram.

K3s packages the common cluster components

Current K3s documentation lists a batteries-included set of components including:

• containerd / compatible CRI runtime support;
• Flannel CNI networking;
• CoreDNS;
• Traefik ingress controller;
• ServiceLB;
• kube-router network-policy controller;
• local-path-provisioner;
• metrics-server;
• Helm controller;
• host utilities required by the distribution.

The exact packaged set can evolve with releases, so production documentation should follow the current K3s release rather than an old component list copied from a tutorial.

These defaults reduce initial integration work, but they are not mandatory architectural choices. K3s allows operators to disable packaged components and replace them where the workload requires a different implementation.

A useful rule is:

Keep the default component until a documented requirement justifies replacing it.

Replacing Traefik, Flannel, ServiceLB, or local-path storage introduces a new operational dependency. That can be the correct decision, but it should solve a real compatibility, networking, security, storage, or platform requirement.

Cloud-VM networking should be private by default

K3s requires specific node-to-node and node-to-server communication.

The current K3s requirements document states that:

• TCP 6443 must be reachable from nodes to the K3s server/API;
• Flannel VXLAN commonly uses UDP 8472 between nodes;
• metrics-server requires node reachability on TCP 10250;
• embedded-etcd server nodes require TCP 2379–2380 between one another.

K3s explicitly warns that the Flannel VXLAN port must not be exposed to the public internet.

For cloud VMs, a safer baseline is therefore:

Internet
   ↓
Public application ingress only
   ↓
Private VM network
   ├── server-to-server control traffic
   ├── agent-to-server API traffic
   ├── pod/network overlay traffic
   └── datastore traffic

Avoid assigning public exposure to every node merely because every VM has a public interface available.

Administrative SSH, Kubernetes API access, datastore access, and overlay networking should each have an explicit policy.

The deeper decisions around Flannel, ingress, public endpoints, private networks, and load balancing belong in K3s Networking, Ingress, and Load Balancing.

Ingress and ServiceLB solve different traffic problems

K3s includes Traefik by default as an ingress controller and ServiceLB as a controller for Kubernetes LoadBalancer Services.

Those are related but different layers.

For a small VM-based cluster, Traefik may be enough for HTTP routing. A more complex production environment may replace it with another ingress or Gateway implementation.

The architectural question is not “which ingress is best?” It is where public traffic enters, how unhealthy backends are removed, where TLS terminates, and which node failure can interrupt traffic.

Those decisions become especially important once the cluster has multiple agents or separate control-plane nodes.

Local-path storage is convenient, not automatically highly available

K3s packages local-path-provisioner as a simple PersistentVolume implementation.

That is convenient because a workload can request persistent storage without first integrating a separate storage platform.

But local storage is tied to the node that owns the underlying path.

Pod
  ↓
PVC
  ↓
Local-path volume
  ↓
Specific node disk

If the Pod is rescheduled to another node, Kubernetes scheduling constraints and the storage implementation determine whether that data remains usable.

Therefore, do not infer the following from the word “PersistentVolume”:

• the data is replicated;
• the data survives node loss;
• the volume can move automatically to another VM;
• the volume is backed up;
• the application is highly available.

For stateful production workloads, choose storage based on the failure you need to survive.

The dedicated K3s Persistent Storage and Backup Strategy owns that recovery design.

K3s server sizing must include workload capacity separately

K3s has lower resource requirements than many manually assembled Kubernetes distributions, but “lightweight” does not mean resource-free.

Current K3s requirements list 2 CPU cores and 2 GB RAM as the minimum for a server node before meaningful workload capacity is added. SSD-backed disks are recommended because datastore performance matters.

That minimum is a bootstrap floor, not a production sizing recommendation.

A single-server K3s VM may need capacity for:

Kubernetes control plane
+ datastore
+ container runtime
+ system Pods
+ ingress
+ monitoring
+ application Pods
+ headroom for rescheduling and upgrades

An agent-only VM removes most control-plane overhead, but it still needs system resources and headroom for workload spikes.

At Raff, we size K3s control capacity and application capacity as separate budgets. If a workload needs 4 vCPU of application capacity, putting it on a 4-vCPU server node leaves no sensible allowance for the cluster services that also run there.

That is why small clusters should reserve headroom even when average CPU utilization looks low.

Server nodes can run workloads, but role separation can improve operations

By default, K3s server nodes are also schedulable nodes. They can run normal application Pods.

For a small cluster, that is efficient:

3 K3s servers
  = control plane + etcd + workloads

For larger or more sensitive clusters, teams may prefer to keep application workloads primarily on agent nodes:

3 K3s servers
  = control plane + etcd

N agents
  = application workloads

Role separation can help when:

• application spikes should not compete with etcd or API-server resources;
• control-plane maintenance should be easier to reason about;
• security boundaries differ;
• workload node pools need different sizes;
• stateful workloads require dedicated placement.

It also costs more because the control-plane VMs consume capacity even when they are not serving application traffic.

Choose the separation because it reduces a specific operational risk, not because every diagram of Kubernetes shows dedicated control-plane nodes.

Storage, HA, and backup are three different controls

K3s production designs often confuse availability with recovery.

A useful distinction is:

An HA cluster can faithfully preserve a bad configuration, an accidental namespace deletion, or corrupted application data.

Backups preserve a recoverable past. HA preserves service through selected present-time failures.

The cluster recovery plan should therefore include both:

1. cluster state — K3s datastore and configuration;
2. application state — databases, PVC data, object data, secrets, and external dependencies.

For broader Kubernetes recovery layers, use Kubernetes Backup and Disaster Recovery Strategy.

Operations ownership is the main cost of self-managed K3s

K3s reduces the amount of cluster assembly. It does not make the cluster self-operating.

A production owner still needs to manage:

• K3s version lifecycle;
• Kubernetes version compatibility;
• server and agent OS patching;
• certificates and tokens;
• firewall rules;
• cluster datastore health;
• node capacity;
• CNI health;
• ingress health;
• storage health;
• backups and restore tests;
• workload resource requests and limits;
• node drains and maintenance;
• incident response.

That ownership is what separates self-managed K3s from managed Kubernetes.

The existing K3s vs Kubernetes guide owns the platform-choice comparison. The practical point here is simpler: K3s reduces integration surface, but the team still operates the control plane.

If operating that control plane does not create product value, managed Kubernetes may be the more appropriate boundary.

Upgrades should be treated as cluster maintenance

K3s tracks Kubernetes releases through K3s-specific builds. As of August 2026, K3s publishes maintained release channels including current v1.35 builds.

Production upgrades should not be treated like updating a normal stateless package on every VM simultaneously.

A safer architecture has:

• a documented supported-version path;
• a current datastore backup;
• enough capacity to drain or restart nodes safely;
• workload disruption budgets appropriate to the application;
• a tested rollback or recovery path;
• staged server and agent upgrades;
• post-upgrade workload validation.

For HA clusters, quorum must remain intact while server nodes are maintained.

The dedicated K3s Upgrade and Rollback Strategy owns the exact maintenance model so this connector guide does not become a procedural upgrade tutorial.

The decision framework matches topology to workload risk

There is no universal “production K3s topology.”

The architecture should follow failure tolerance, recovery objectives, workload state, team skill, and maintenance capacity.

A three-server embedded-etcd cluster is not automatically better than one server if nobody can maintain quorum or restore etcd. A single server is not automatically unsafe if the workload is replaceable and the recovery window is acceptable.

The correct design is the smallest topology whose failure and recovery behavior the team can explain before an incident.

Raff VMs provide the self-managed K3s boundary

K3s can run on standard Linux cloud VMs, which makes Raff VM the natural Raff infrastructure boundary for a self-managed deployment.

A staged Raff topology can look like:

Stage 1
Raff VM
  └── single K3s server

Stage 2
Raff VM server
  ├── Raff VM agent
  └── Raff VM agent

Stage 3
3 Raff VM K3s servers
  + agent VMs as required
  + private node communication
  + externalized application data where appropriate

Keep cluster communication on private networking wherever possible and expose only the endpoints the architecture actually requires.

Use Raff VM when your team deliberately wants to own K3s servers, agents, upgrades, datastore recovery, networking, and cluster maintenance.

Use Raff Kubernetes when the Kubernetes API and workload model are valuable but self-managing the control plane is not.

At Raff, the useful boundary is responsibility rather than branding: self-managed K3s gives the team more cluster ownership; managed Kubernetes removes some of that ownership.

Production architecture checklist

Before calling a K3s-on-VM design production-ready, confirm:

• Single-server or HA topology is explicitly chosen.
• Server and agent roles are documented.
• Control-plane and workload capacity are sized separately.
• Server disks use appropriate low-latency storage.
• API port 6443 is restricted to required sources.
• Flannel overlay traffic stays on trusted/private networks.
• Embedded-etcd ports are restricted to server nodes when used.
• A fixed registration endpoint exists for HA where required.
• Ingress and application traffic entry are documented separately from the API endpoint.
• Persistent workloads have an explicit storage class and failure model.
• Cluster-state backup is separate from application-data backup.
• Secrets and node tokens have controlled recovery paths.
• Monitoring covers server, datastore, node, and workload health.
• Upgrade order and maintenance capacity are documented.
• Datastore and application restores have been tested.
• One team or person owns after-hours cluster incidents.

K3s on cloud VMs is a self-managed Kubernetes architecture

K3s reduces Kubernetes packaging and integration work, but the architecture still has the same core production questions: control-plane availability, datastore quorum, node networking, workload placement, ingress, persistent storage, backups, upgrades, and recovery.

Start with one server when the workload can tolerate that failure boundary. Add agents when compute capacity is the problem. Move to three or more embedded-etcd servers, or an external-datastore HA design, when control-plane availability justifies the extra operating surface.

Most importantly, keep availability, persistence, and recovery separate. A cluster that survives one node failure is not automatically a cluster that can recover deleted application data or corrupted state.

Continue with K3s High Availability: Embedded etcd vs External Database for datastore topology, then K3s Networking, Ingress, and Load Balancing for the VM network boundary.

Sources

• K3s Architecture
• K3s Requirements
• K3s Quick-Start Guide
• K3s Cluster Datastore
• K3s High Availability with Embedded etcd
• K3s Cluster Load Balancer
• K3s Server Configuration
• Documentation verified: 2026-08-23