The adoption of Talos OS as an operating system for Kubernetes nodes represents a paradigm shift towards immutability, security, and declarative management via API. However, the minimalist nature and the lack of a traditional shell in Talos pose specific challenges when it comes to configuring the high availability (HA) endpoint for the API server and exposing services to the outside. The choice between the native Talos Virtual IP (VIP), kube-vip, and MetalLB is not purely technical, but depends on the cluster scale, latency requirements, and the complexity of the underlying network infrastructure.1 A deep understanding of how these components interact with the Linux kernel and the Kubernetes control plane is essential to implement a load balancing strategy that is resilient and scalable.
Fundamentals of Control Plane High Availability in Talos OS#
The heart of a Kubernetes cluster is its control plane, which includes critical components such as etcd, kube-apiserver, kube-scheduler, and kube-controller-manager. In Talos OS, these components are executed as static pods managed directly by the kubelet.5 The main challenge in the architecture of an HA cluster consists of providing clients, such as kubectl or worker nodes, with a single stable endpoint (an IP address or a URL) that can reach any available control plane node, ensuring operational continuity even in the event of failure of one or more nodes.1
Talos OS addresses this challenge through different methodologies, each with different implications in terms of failover speed and load capacity. The most immediate approach is the use of the native VIP integrated into the operating system, but as the external load on the API server increases, the need emerges for more sophisticated solutions such as external load balancers or BGP-based implementations.7
The Mechanism of the Native Talos Virtual IP#
The native Talos VIP is a built-in feature designed to simplify the creation of HA clusters without requiring external resources like reverse proxies or hardware load balancers.1 This mechanism relies on the contention of the shared IP address among control plane nodes through an election process managed by etcd.1
From an operational perspective, the configuration requires that all control plane nodes share a Layer 2 network. The VIP address must be a reserved address and not used within the same subnet as the nodes.1 A crucial aspect of this implementation is that the VIP does not become active until the Kubernetes cluster has been bootstrapped, since its management depends directly on the health state of etcd.1
| Native VIP Characteristic | Technical Detail |
|---|---|
| Network Requirement | Layer 2 Connectivity (same subnet/switch) |
| Election Mechanism | Based on etcd quorum |
| Failover Behavior | Almost instant for graceful shutdowns; up to 1 minute for sudden crashes |
| Load Limitation | Only one node receives traffic at a time (Active-Passive) |
| Bootstrap Dependency | Active only after etcd cluster formation |
1
The analysis of failover times reveals an important design decision by the creators of Talos. While an orderly disconnection allows for an immediate handover, a sudden failure requires Talos to wait for the etcd election timeout. This delay is intentional and serves to ensure that “split-brain” scenarios do not occur, where multiple nodes announce the same IP simultaneously, a situation that could corrupt network sessions and destabilize access to the API.1
KubePrism: The Silent Hero of Internal High Availability#
Often confused with external VIP solutions, KubePrism is actually a complementary and distinct feature.8 While the native VIP or kube-vip serve primarily for external access (such as kubectl commands), KubePrism is designed exclusively for internal access to the cluster.7 It creates a local load balancing endpoint on every node of the cluster (usually on localhost:7445), which internal processes like the kubelet use to communicate with the API server.8
The importance of KubePrism lies in its ability to abstract the complexity of the control plane from the worker nodes. If the external load balancer or the VIP were to fail, KubePrism has an automatic fallback mechanism that allows nodes to continue operating by communicating directly with the control plane nodes.7 In production architectures, it is recommended to keep KubePrism always enabled to ensure that the internal health of the cluster never depends solely on a single external network endpoint.7
Analysis of Strategies for Service Load Balancing#
Besides access to the API server, managing traffic towards workloads requires the implementation of services of type LoadBalancer. In bare-metal or virtualized environments where Talos is commonly distributed, this functionality is not automatically provided by the cloud provider, making it necessary to install specific controllers like MetalLB or kube-vip.3
MetalLB: The Standard for Bare-Metal Services#
MetalLB is likely the most mature and widespread solution for providing load balancing in on-premise environments.3 It operates by monitoring resources of type Service with spec.type: LoadBalancer and assigning them an IP address from a preconfigured pool.3
MetalLB supports two main operating modes: Layer 2 and BGP. In Layer 2 mode, one of the cluster nodes is elected “leader” for a given service IP address and responds to ARP requests (for IPv4) or NDP (for IPv6).3 Although extremely simple to configure, this mode presents the limitation of funneling all traffic of a service through a single node, creating a potential bottleneck.4 Conversely, BGP mode allows each node to announce the service IP address to network routers, enabling true load balancing via ECMP (Equal-Cost Multi-Path).4
Kube-vip: Versatility and Unification#
Kube-vip stands out for its ability to manage both control plane HA and service load balancing in a single component.2 Unlike the native Talos VIP, kube-vip can be configured to use IPVS (IP Virtual Server) to distribute API server traffic across all control plane nodes in active-active mode, significantly improving performance under high load.14
Kube-vip can run as a static pod, making it ideal for scenarios where the HA endpoint must be available from the very first moments of the cluster bootstrap, even before the etcd database is fully formed.14 However, its configuration as a service load balancer is often considered less feature-rich compared to MetalLB, which offers more granular management of address pools and advertisement policies.16
Comparison of Strategies Requested by the User#
Choosing the correct combination of tools depends on the need to balance operational simplicity and scalability. Below is an analysis of the comparison between the three main strategies raised in the query.
Strategy 1: Native Talos VIP with MetalLB#
This is the most common and recommended configuration for small to medium-sized clusters (up to 10-20 nodes) in Layer 2 environments.7
- Advantages: Leverages operating system stability for critical API access and uses MetalLB, which is the industry standard, for application service management. The separation of duties makes the system easy to diagnose: API issues are linked to Talos configuration, while application issues are linked to MetalLB.17
- Disadvantages: Access to the API server is limited to the capacity of a single node (active-passive), which may not be sufficient for clusters with a very high frequency of API operations (e.g., massive CI/CD environments).7
Strategy 2: Kube-vip without MetalLB#
This strategy aims at unifying network functions under a single controller.2
- Advantages: Reduces the number of components to manage in the cluster. Kube-vip can manage both the API server IP and LoadBalancer service IPs. Supports IPVS for real API balancing.14
- Disadvantages: Although versatile, kube-vip can result in being more complex to configure correctly to cover all MetalLB use cases, especially in complex BGP networks. The loss of the kube-vip pod could, in theory, interrupt both access to the control plane and all cluster services simultaneously.16
Strategy 3: Kube-vip with MetalLB#
In this configuration, kube-vip is used exclusively for control plane high availability, while MetalLB manages application services.16
- Advantages: Offers the best performance for the API server (thanks to IPVS or BGP ECMP provided by kube-vip) while maintaining the flexibility of MetalLB for applications.17 It is an excellent choice for enterprise environments where the control plane is under heavy stress.
- Disadvantages: It is the most complex configuration to maintain, requiring the management of two different network controllers that could conflict if not carefully configured (for example, both attempting to listen on BGP port 179).3
| Characteristic | Native VIP + MetalLB | Kube-vip (Only) | Kube-vip + MetalLB |
|---|---|---|---|
| Complexity | Low | Medium | High |
| API Performance | Active-Passive | Active-Active (IPVS) | Active-Active (IPVS) |
| Service Performance | High (L2/BGP) | Medium | High (L2/BGP) |
| Standardization | Very Common | Common | Professional/Enterprise |
| Recommended Use | Homelab / SMB | Minimalist Systems | High Load Clusters |
3
Differentiated Strategies by Cluster Size#
Cluster sizing is a determining factor for choosing the balancing strategy. What works for a small home server might not be adequate for a distributed data center.
Small Clusters and “Minecraft” Environments#
By “Minecraft configuration” we usually mean a small-sized cluster, often consisting of a single node or a small set of nodes (3 or less), typical of homelab or test environments.21
In a single-node cluster, it is fundamental to pay attention to a technical detail of Talos: by default, control plane nodes are labeled to be excluded from external load balancers (node.kubernetes.io/exclude-from-external-load-balancers: “”).24 In a multi-node cluster, this protects master nodes from application traffic, but in a single-node cluster, it prevents MetalLB or kube-vip from correctly exposing services.24 The solution consists of removing or commenting out this label in the machine configuration.24
For these small clusters, the recommendation is absolute simplicity:
- Control Plane: Use the native Talos VIP.7
- Services: Use MetalLB in Layer 2 mode.10
- Storage: Often coupled with Longhorn for simplicity of management on few nodes.7
Large Clusters (>100 Nodes)#
In enterprise-scale clusters, Layer 2 network limitations become evident. ARP broadcast traffic for VIP management can degrade network performance, and failover speed based on etcd election might not meet availability requirements.4
Guidelines from Sidero Labs (the developers of Talos) for high-load clusters suggest moving the responsibility of API server balancing outside the cluster.6 The use of an external load balancer (F5, Netscaler, or a dedicated HAProxy instance) that distributes requests to all healthy control plane nodes is the most resilient option.6 This approach offloads the master nodes’ CPU from network traffic management and ensures that API access is independent of the internal state of the Kubernetes control plane.7
For services, at this scale, the use of BGP mode is imperative.4 MetalLB or Cilium (which offers a native eBPF-based BGP control plane) become the tools of choice.18 Integration with TOR (Top of Rack) routers allows for a truly horizontal traffic distribution, leveraging the physical network infrastructure to ensure scalability.27
Technical Analysis of Protocols: ARP vs BGP#
The decision between Layer 2 (ARP) and Layer 3 (BGP) is dictated by infrastructure. It is fundamental to understand the “cost” of each choice.
Implications of Layer 2 and ARP#
ARP-based balancing is fundamentally a failover mechanism, not load distribution.12 When MetalLB or kube-vip operate in this mode, they choose one node that responds to all requests for a given IP.3 The advantage is that it works everywhere, even on cheap switches.29 However, in case of leader node failure, a “gratuitous” ARP packet must be sent to inform other hosts that the MAC address associated with that IP has changed.12 If clients or network routers have persistent ARP caches and ignore gratuitous advertisements, connectivity interruptions of up to 30-60 seconds can occur.12
Implications of Layer 3 and BGP#
BGP transforms Kubernetes nodes into actual routers.13 Each node announces service IP prefixes to a BGP peer (usually the default gateway). This allows ECMP balancing, where the router distributes packets among nodes.4
However, BGP on Kubernetes presents a challenge known as connection “churn”. Since traditional routers are often “stateless” in their ECMP hashing, when a node is added or removed (e.g., during a Talos upgrade), the router’s hashing algorithm might recalculate paths, moving active TCP sessions to different nodes.13 If the new node does not know that session (because traffic was not proxied correctly), the connection will be interrupted with a “Connection Reset” error.13 To overcome this, it is necessary to use routers that support “Resilient ECMP” or place services behind an Ingress controller that can manage session persistence at the application level.13
Configuration Guide: Details and Warnings#
Configuring these strategies in Talos OS requires the use of YAML patches applied to the machine configuration files (machineconfig).
Configuring the Native VIP#
A common mistake is using the VIP as an endpoint in the talosconfig file.1 Since the VIP depends on the health of etcd and the kube-apiserver, if these components fail, it will not be possible to use talosctl via the VIP to repair the node. The correct practice involves inserting the individual physical IP addresses of the master nodes in the endpoint list of talosconfig.6
Conflicts between Kube-vip and MetalLB#
If choosing to use kube-vip for the control plane and MetalLB for services, it is vital to use load balancing classes (loadBalancerClass) introduced in Kubernetes 1.24.17 Without this distinction, both controllers might attempt to “take charge” of the same service, leading to a situation of instability where the IP address is continuously assigned and removed.17
Furthermore, if both components are configured to use BGP, they are likely to conflict over the use of TCP port 179.3 In Talos, a modern solution consists of using Cilium as CNI and entrusting it with the entire BGP control plane, eliminating the need for MetalLB and reducing system complexity.18
Special Use Cases and Troubleshooting#
In real installations, undocumented scenarios often emerge requiring specific interventions.
Asymmetric Routing Issues#
When using software load balancers on Talos, the phenomenon of asymmetric routing can occur: the packet enters the cluster via node A (which holds the VIP) but must be delivered to a pod on node B.32 If node B responds directly to the client via its own default gateway, many firewalls will block the traffic considering it an attack or a protocol error.32
To mitigate this issue, Talos and MetalLB recommend enabling “strict ARP” mode in kube-proxy.31 This ensures that traffic follows predictable paths. Another option is the use of externalTrafficPolicy: Local in the Kubernetes service, which instructs the load balancer to send traffic only to nodes that actually host the service pod, eliminating the internal hop between nodes and preserving the client’s source IP address.13
Failover and Impact on Workloads#
It is fundamental to understand that VIP failover (whether native or managed by kube-vip) affects only external access to the cluster (e.g., executing kubectl or external API calls).1 Inside the cluster, thanks to KubePrism and service discovery, workloads continue to communicate normally and are unaffected by the state of the external VIP.1 However, long-lived connections passing through the VIP (such as gRPC tunnels or HTTP/2 sessions) will be interrupted and require client-side reconnection logic.1
Conclusions and Strategic Recommendations#
Based on the analysis of collected data and industry best practices, the correct strategy for a Kubernetes cluster on Talos OS can be summarized in three main paths, depending on scalability needs and network complexity.
For most users, the Recommended Strategy is the pairing of the native Talos VIP with MetalLB in Layer 2 mode. This configuration perfectly balances the management simplicity typical of Talos with the flexibility of MetalLB. It is the ideal choice for clusters operating in a single server room or standard virtualized environment, ensuring high availability of the API server without adding critical components that must be manually managed during bootstrap.
For Enterprise or High Load installations, the optimal strategy shifts towards external load balancing for the control plane and MetalLB or Cilium in BGP mode for services. This architecture eliminates the typical bottlenecks of Layer 2 networks and leverages the power of physical routers to distribute traffic, ensuring that the cluster can scale up to hundreds of nodes without network performance degradation.
Finally, for Small Clusters and Homelabs (Minecraft Style), the watchword is minimalism. The use of native VIP and MetalLB (L2), taking care to correctly configure node labels to allow service exposure, provides a robust and easy-to-maintain environment, minimizing the “consumption” of precious resources by infrastructure components.
In summary, the systems architect operating with Talos OS must always prioritize enabling KubePrism as the foundation of internal resilience and select the IP address advertisement method (ARP vs BGP) not based on software preference, but based on the actual capabilities of the network hardware hosting the cluster.
Bibliography#
- Virtual (shared) IP - Sidero Documentation - What is Talos Linux?, accessed on January 1, 2026, https://docs.siderolabs.com/talos/v1.8/networking/vip
- kube-vip: Documentation, accessed on January 1, 2026, https://kube-vip.io/
- Setting Up MetalLB: Kubernetes LoadBalancer for Bare Metal Clusters | Talha Juikar, accessed on January 1, 2026, https://talhajuikar.com/posts/metallb/
- MetalLB: A Load Balancer for Bare Metal Kubernetes Clusters | by 8grams - Medium, accessed on January 1, 2026, https://8grams.medium.com/metallb-a-load-balancer-for-bare-metal-kubernetes-clusters-ef8a9e00c2bd
- Control Plane - Sidero Documentation - What is Talos Linux?, accessed on January 1, 2026, https://docs.siderolabs.com/talos/v1.9/learn-more/control-plane
- Production Clusters - Sidero Documentation - What is Talos Linux?, accessed on January 1, 2026, https://docs.siderolabs.com/talos/v1.7/getting-started/prodnotes
- Kubernetes Cluster Reference Architecture with Talos Linux for 2025-05 - Sidero Labs, accessed on January 1, 2026, https://www.siderolabs.com/wp-content/uploads/2025/08/Kubernetes-Cluster-Reference-Architecture-with-Talos-Linux-for-2025-05.pdf
- difference VIP vs KubePrism (or other) · siderolabs talos · Discussion #9906 - GitHub, accessed on January 1, 2026, https://github.com/siderolabs/talos/discussions/9906
- Installation - kube-vip, accessed on January 1, 2026, https://kube-vip.io/docs/installation/
- Kubernetes Homelab Series Part 3 - LoadBalancer With MetalLB | Eric Daly’s Blog, accessed on January 1, 2026, https://blog.dalydays.com/post/kubernetes-homelab-series-part-3-loadbalancer-with-metallb/
- Configuration :: MetalLB, bare metal load-balancer for Kubernetes, accessed on January 1, 2026, https://metallb.universe.tf/configuration/
- MetalLB in layer 2 mode :: MetalLB, bare metal load-balancer for Kubernetes, accessed on January 1, 2026, https://metallb.universe.tf/concepts/layer2/
- MetalLB in BGP mode :: MetalLB, bare metal load-balancer for Kubernetes, accessed on January 1, 2026, https://metallb.universe.tf/concepts/bgp/
- Architecture | kube-vip, accessed on January 1, 2026, https://kube-vip.io/docs/about/architecture/
- Static Pods | kube-vip, accessed on January 1, 2026, https://kube-vip.io/docs/installation/static/
- What do you use for baremetal VIP ControlPane and Services : r/kubernetes - Reddit, accessed on January 1, 2026, https://www.reddit.com/r/kubernetes/comments/1nlnb1o/what_do_you_use_for_baremetal_vip_controlpane_and/
- HA Kubernetes API server with MetalLB…? - Reddit, accessed on January 1, 2026, https://www.reddit.com/r/kubernetes/comments/1o9t1j2/ha_kubernetes_api_server_with_metallb/
- For those who work with HA onprem clusters : r/kubernetes - Reddit, accessed on January 1, 2026, https://www.reddit.com/r/kubernetes/comments/1j05ozt/for_those_who_work_with_ha_onprem_clusters/;
- Kubernetes Load-Balancer service - kube-vip, accessed on January 1, 2026, https://kube-vip.io/docs/usage/kubernetes-services/
- metallb + BGP = conflict with kube-router? | TrueNAS Community, accessed on January 1, 2026, https://www.truenas.com/community/threads/metallb-bgp-conflict-with-kube-router.115690/
- Talos Kubernetes in Five Minutes - DEV Community, accessed on January 1, 2026, https://dev.to/nabsul/talos-kubernetes-in-five-minutes-1p1h
- [Lab Setup] 3-node Talos cluster (Mac minis) + MinIO backend — does this topology make sense? : r/kubernetes - Reddit, accessed on January 1, 2026, https://www.reddit.com/r/kubernetes/comments/1myb8xc/lab_setup_3node_talos_cluster_mac_minis_minio/
- Getting back into the HomeLab game for 2024 - vZilla, accessed on January 1, 2026, https://vzilla.co.uk/vzilla-blog/getting-back-into-the-homelab-game-for-2024
- Fix LoadBalancer Services Not Working on Single Node Talos Kubernetes Cluster, accessed on January 1, 2026, https://www.robert-jensen.dk/posts/2025/fix-loadbalancer-services-not-working-on-single-node-talos-kubernetes-cluster/
- Deploy Talos Linux with Local VIP, Tailscale, Longhorn, MetalLB and Traefik - Josh’s Notes, accessed on January 1, 2026, https://notes.joshrnoll.com/notes/deploy-talos-linux-with-local-vip-tailscale-longhorn-metallb-and-traefik/
- Kubernetes & Talos - Reddit, accessed on January 1, 2026, https://www.reddit.com/r/kubernetes/comments/1hs6bui/kubernetes_talos/
- Advanced BGP configuration :: MetalLB, bare metal load-balancer for Kubernetes, accessed on January 1, 2026, https://metallb.universe.tf/configuration/_advanced_bgp_configuration/
- Talos with redundant routed networks via bgp : r/kubernetes - Reddit, accessed on January 1, 2026, https://www.reddit.com/r/kubernetes/comments/1iy411r/talos_with_redundant_routed_networks_via_bgp/
- MetalLB on K3s (using Layer 2 Mode) | SUSE Edge Documentation, accessed on January 1, 2026, https://documentation.suse.com/suse-edge/3.3/html/edge/guides-metallb-k3s.html
- Troubleshooting - Sidero Documentation - What is Talos Linux?, accessed on January 1, 2026, https://docs.siderolabs.com/talos/v1.9/troubleshooting/troubleshooting
- Installation :: MetalLB, bare metal load-balancer for Kubernetes, accessed on January 1, 2026, https://metallb.universe.tf/installation/
- Analyzing Load Balancer VIP Routing with Calico BGP and MetalLB - AHdark Blog, accessed on January 1, 2026, https://www.ahdark.blog/analyzing-load-balancer-vip-routing/
- Kubernetes Services : Achieving optimal performance is elusive | by CloudyBytes | Medium, accessed on January 1, 2026, https://cloudybytes.medium.com/kubernetes-services-achieving-optimal-performance-is-elusive-5def5183c281
- Usage :: MetalLB, bare metal load-balancer for Kubernetes, accessed on January 1, 2026, https://metallb.universe.tf/usage/