diff --git a/content/en/docs/v1/operations/multi-location/_index.md b/content/en/docs/v1/operations/multi-location/_index.md new file mode 100644 index 00000000..3680c93d --- /dev/null +++ b/content/en/docs/v1/operations/multi-location/_index.md @@ -0,0 +1,15 @@ +--- +title: "Multi-Location Clusters" +linkTitle: "Multi-Location" +description: "Extend Cozystack management clusters across multiple locations using Kilo WireGuard mesh, cloud autoscaling, and local cloud controller manager." +weight: 40 +--- + +This section covers extending a Cozystack management cluster across multiple physical locations +(on-premises + cloud, multi-cloud, etc.) using WireGuard mesh networking. + +The setup consists of three components: + +- [Networking Mesh]({{% ref "networking-mesh" %}}) -- Kilo WireGuard mesh with Cilium IPIP encapsulation +- [Local CCM]({{% ref "local-ccm" %}}) -- cloud controller manager for node IP detection and lifecycle +- [Cluster Autoscaling]({{% ref "autoscaling" %}}) -- automatic node provisioning in cloud providers diff --git a/content/en/docs/v1/operations/multi-location/autoscaling/_index.md b/content/en/docs/v1/operations/multi-location/autoscaling/_index.md new file mode 100644 index 00000000..2a03ed0d --- /dev/null +++ b/content/en/docs/v1/operations/multi-location/autoscaling/_index.md @@ -0,0 +1,19 @@ +--- +title: "Cluster Autoscaling" +linkTitle: "Autoscaling" +description: "Automatic node scaling for Cozystack management clusters using Kubernetes Cluster Autoscaler." +weight: 20 +--- + +The `cluster-autoscaler` system package enables automatic node scaling for Cozystack management clusters. +It monitors pending pods and automatically provisions or removes cloud nodes based on demand. + +Before configuring autoscaling, complete the [Networking Mesh]({{% ref "../networking-mesh" %}}) +and [Local CCM]({{% ref "../local-ccm" %}}) setup. + +Cozystack provides pre-configured variants for different cloud providers: + +- [Hetzner Cloud]({{% ref "hetzner" %}}) -- scale using Hetzner Cloud servers +- [Azure]({{% ref "azure" %}}) -- scale using Azure Virtual Machine Scale Sets + +Each variant is deployed as a separate Cozystack Package with provider-specific configuration. diff --git a/content/en/docs/v1/operations/multi-location/autoscaling/azure.md b/content/en/docs/v1/operations/multi-location/autoscaling/azure.md new file mode 100644 index 00000000..94207185 --- /dev/null +++ b/content/en/docs/v1/operations/multi-location/autoscaling/azure.md @@ -0,0 +1,424 @@ +--- +title: "Cluster Autoscaler for Azure" +linkTitle: "Azure" +description: "Configure automatic node scaling in Azure with Talos Linux and VMSS." +weight: 20 +--- + +This guide explains how to configure cluster-autoscaler for automatic node scaling in Azure with Talos Linux. + +## Prerequisites + +- Azure subscription with Contributor Service Principal +- `az` CLI installed +- Existing Talos Kubernetes cluster +- [Networking Mesh]({{% ref "../networking-mesh" %}}) and [Local CCM]({{% ref "../local-ccm" %}}) configured + +## Step 1: Create Azure Infrastructure + +### 1.1 Login with Service Principal + +```bash +az login --service-principal \ + --username "" \ + --password "" \ + --tenant "" +``` + +### 1.2 Create Resource Group + +```bash +az group create \ + --name \ + --location +``` + +### 1.3 Create VNet and Subnet + +```bash +az network vnet create \ + --resource-group \ + --name cozystack-vnet \ + --address-prefix 10.2.0.0/16 \ + --subnet-name workers \ + --subnet-prefix 10.2.0.0/24 \ + --location +``` + +### 1.4 Create Network Security Group + +```bash +az network nsg create \ + --resource-group \ + --name cozystack-nsg \ + --location + +# Allow WireGuard +az network nsg rule create \ + --resource-group \ + --nsg-name cozystack-nsg \ + --name AllowWireGuard \ + --priority 100 \ + --direction Inbound \ + --access Allow \ + --protocol Udp \ + --destination-port-ranges 51820 + +# Allow Talos API +az network nsg rule create \ + --resource-group \ + --nsg-name cozystack-nsg \ + --name AllowTalosAPI \ + --priority 110 \ + --direction Inbound \ + --access Allow \ + --protocol Tcp \ + --destination-port-ranges 50000 + +# Associate NSG with subnet +az network vnet subnet update \ + --resource-group \ + --vnet-name cozystack-vnet \ + --name workers \ + --network-security-group cozystack-nsg +``` + +## Step 2: Create Talos Image + +### 2.1 Generate Schematic ID + +Create a schematic at [factory.talos.dev](https://factory.talos.dev) with required extensions: + +```bash +curl -s -X POST https://factory.talos.dev/schematics \ + -H "Content-Type: application/json" \ + -d '{ + "customization": { + "systemExtensions": { + "officialExtensions": [ + "siderolabs/amd-ucode", + "siderolabs/amdgpu-firmware", + "siderolabs/bnx2-bnx2x", + "siderolabs/drbd", + "siderolabs/i915-ucode", + "siderolabs/intel-ice-firmware", + "siderolabs/intel-ucode", + "siderolabs/qlogic-firmware", + "siderolabs/zfs" + ] + } + } + }' +``` + +Save the returned `id` as `SCHEMATIC_ID`. + +### 2.2 Create Managed Image from VHD + +```bash +# Download Talos Azure image +curl -L -o azure-amd64.raw.xz \ + "https://factory.talos.dev/image/${SCHEMATIC_ID}//azure-amd64.raw.xz" + +# Decompress +xz -d azure-amd64.raw.xz + +# Convert to VHD +qemu-img convert -f raw -o subformat=fixed,force_size -O vpc \ + azure-amd64.raw azure-amd64.vhd + +# Get VHD size +VHD_SIZE=$(stat -f%z azure-amd64.vhd) # macOS +# VHD_SIZE=$(stat -c%s azure-amd64.vhd) # Linux + +# Create managed disk for upload +az disk create \ + --resource-group \ + --name talos- \ + --location \ + --upload-type Upload \ + --upload-size-bytes $VHD_SIZE \ + --sku Standard_LRS \ + --os-type Linux \ + --hyper-v-generation V2 + +# Get SAS URL for upload +SAS_URL=$(az disk grant-access \ + --resource-group \ + --name talos- \ + --access-level Write \ + --duration-in-seconds 3600 \ + --query accessSAS --output tsv) + +# Upload VHD +azcopy copy azure-amd64.vhd "$SAS_URL" --blob-type PageBlob + +# Revoke access +az disk revoke-access \ + --resource-group \ + --name talos- + +# Create managed image from disk +az image create \ + --resource-group \ + --name talos- \ + --location \ + --os-type Linux \ + --hyper-v-generation V2 \ + --source $(az disk show --resource-group \ + --name talos- --query id --output tsv) +``` + +## Step 3: Create Talos Machine Config for Azure + +From your cluster repository, generate a worker config file: + +```bash +talm template -t templates/worker.yaml --offline --full > nodes/azure.yaml +``` + +Then edit `nodes/azure.yaml` for Azure workers: + +1. Add Azure location metadata (see [Networking Mesh]({{% ref "../networking-mesh" %}})): + ```yaml + machine: + nodeAnnotations: + kilo.squat.ai/location: azure + kilo.squat.ai/persistent-keepalive: "20" + nodeLabels: + topology.kubernetes.io/zone: azure + ``` +2. Set public Kubernetes API endpoint: + Change `cluster.controlPlane.endpoint` to the **public** API server address (for example `https://:6443`). You can find this address in your kubeconfig or publish it via ingress. +3. Remove discovered installer/network sections: + Delete `machine.install` and `machine.network` sections from this file. +4. Set external cloud provider for kubelet (see [Local CCM]({{% ref "../local-ccm" %}})): + ```yaml + machine: + kubelet: + extraArgs: + cloud-provider: external + ``` +5. Fix node IP subnet detection: + Set `machine.kubelet.nodeIP.validSubnets` to the actual Azure subnet where autoscaled nodes run (for example `192.168.102.0/23`). +6. (Optional) Add registry mirrors to avoid Docker Hub rate limiting: + ```yaml + machine: + registries: + mirrors: + docker.io: + endpoints: + - https://mirror.gcr.io + ``` + +Result should include at least: + +```yaml +machine: + nodeAnnotations: + kilo.squat.ai/location: azure + kilo.squat.ai/persistent-keepalive: "20" + nodeLabels: + topology.kubernetes.io/zone: azure + kubelet: + nodeIP: + validSubnets: + - 192.168.102.0/23 # replace with your Azure workers subnet + extraArgs: + cloud-provider: external + registries: + mirrors: + docker.io: + endpoints: + - https://mirror.gcr.io +cluster: + controlPlane: + endpoint: https://:6443 +``` + +All other settings (cluster tokens, CA, extensions, etc.) remain the same as the generated template. + +## Step 4: Create VMSS (Virtual Machine Scale Set) + +```bash +IMAGE_ID=$(az image show \ + --resource-group \ + --name talos- \ + --query id --output tsv) + +az vmss create \ + --resource-group \ + --name workers \ + --location \ + --orchestration-mode Uniform \ + --image "$IMAGE_ID" \ + --vm-sku Standard_D2s_v3 \ + --instance-count 0 \ + --vnet-name cozystack-vnet \ + --subnet workers \ + --public-ip-per-vm \ + --custom-data nodes/azure.yaml \ + --security-type Standard \ + --admin-username talos \ + --authentication-type ssh \ + --generate-ssh-keys \ + --upgrade-policy-mode Manual + +# Enable IP forwarding on VMSS NICs (required for Kilo leader to forward traffic) +az vmss update \ + --resource-group \ + --name workers \ + --set virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].enableIPForwarding=true +``` + +{{% alert title="Important" color="warning" %}} +- Must use `--orchestration-mode Uniform` (cluster-autoscaler requires Uniform mode) +- Must use `--public-ip-per-vm` for WireGuard connectivity +- IP forwarding must be enabled on VMSS NICs so the Kilo leader can forward traffic between the WireGuard mesh and non-leader nodes in the same subnet +- Check VM quota in your region: `az vm list-usage --location ` +- `--custom-data` passes the Talos machine config to new instances +{{% /alert %}} + +## Step 5: Deploy Cluster Autoscaler + +Create the Package resource: + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: Package +metadata: + name: cozystack.cluster-autoscaler-azure +spec: + variant: default + components: + cluster-autoscaler-azure: + values: + cluster-autoscaler: + azureClientID: "" + azureClientSecret: "" + azureTenantID: "" + azureSubscriptionID: "" + azureResourceGroup: "" + azureVMType: "vmss" + autoscalingGroups: + - name: workers + minSize: 0 + maxSize: 10 +``` + +Apply: +```bash +kubectl apply -f package.yaml +``` + +## Step 6: Kilo WireGuard Connectivity + +Azure nodes are behind NAT, so their initial WireGuard endpoint will be a private IP. Kilo handles this automatically through WireGuard's built-in NAT traversal when `persistent-keepalive` is configured (already included in the machine config from Step 3). + +The flow works as follows: +1. The Azure node initiates a WireGuard handshake to the on-premises leader (which has a public IP) +2. `persistent-keepalive` sends periodic keepalive packets, maintaining the NAT mapping +3. The on-premises Kilo leader discovers the real public endpoint of the Azure node through WireGuard +4. Kilo stores the discovered endpoint and uses it for subsequent connections + +{{% alert title="Note" color="info" %}} +No manual `force-endpoint` annotation is needed. The `kilo.squat.ai/persistent-keepalive: "20"` annotation in the machine config is sufficient for Kilo to discover NAT endpoints automatically. Without this annotation, Kilo's NAT traversal mechanism is disabled and the tunnel will not stabilize. +{{% /alert %}} + +## Testing + +### Manual scale test + +```bash +# Scale up +az vmss scale --resource-group --name workers --new-capacity 1 + +# Check node joined +kubectl get nodes -o wide + +# Check WireGuard tunnel +kubectl logs -n cozy-kilo + +# Scale down +az vmss scale --resource-group --name workers --new-capacity 0 +``` + +### Autoscaler test + +Deploy a workload to trigger autoscaling: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: test-azure-autoscale +spec: + replicas: 3 + selector: + matchLabels: + app: test-azure + template: + metadata: + labels: + app: test-azure + spec: + nodeSelector: + topology.kubernetes.io/zone: azure + containers: + - name: pause + image: registry.k8s.io/pause:3.9 + resources: + requests: + cpu: "500m" + memory: "512Mi" +``` + +## Troubleshooting + +### Connecting to remote workers for diagnostics + +Talos does not allow opening a dashboard directly to worker nodes. Use `talm dashboard` +to connect through the control plane: + +```bash +talm dashboard -f nodes/.yaml -n +``` + +Where `.yaml` is your control plane node config and `` is +the Kubernetes internal IP of the remote worker. + +### Node doesn't join cluster +- Check that the Talos machine config control plane endpoint is reachable from Azure +- Verify NSG rules allow outbound traffic to port 6443 +- Check VMSS instance provisioning state: `az vmss list-instances --resource-group --name workers` + +### Non-leader nodes unreachable (kubectl logs/exec timeout) + +If `kubectl logs` or `kubectl exec` works for the Kilo leader node but times out for all other nodes in the same Azure subnet: + +1. **Verify IP forwarding** is enabled on the VMSS: + ```bash + az vmss show --resource-group --name workers \ + --query "virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].enableIPForwarding" + ``` + If `false`, enable it and apply to existing instances: + ```bash + az vmss update --resource-group --name workers \ + --set virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].enableIPForwarding=true + az vmss update-instances --resource-group --name workers --instance-ids "*" + ``` + +2. **Test the return path** from the leader node: + ```bash + # This should work (same subnet, direct) + kubectl exec -n cozy-kilo -- ping -c 2 + ``` + +### VM quota errors +- Check quota: `az vm list-usage --location ` +- Request quota increase via Azure portal +- Try a different VM family that has available quota + +### SkuNotAvailable errors +- Some VM sizes may have capacity restrictions in certain regions +- Try a different VM size: `az vm list-skus --location --size ` diff --git a/content/en/docs/v1/operations/multi-location/autoscaling/hetzner.md b/content/en/docs/v1/operations/multi-location/autoscaling/hetzner.md new file mode 100644 index 00000000..4f28ec1b --- /dev/null +++ b/content/en/docs/v1/operations/multi-location/autoscaling/hetzner.md @@ -0,0 +1,420 @@ +--- +title: "Cluster Autoscaler for Hetzner Cloud" +linkTitle: "Hetzner" +description: "Configure automatic node scaling in Hetzner Cloud with Talos Linux." +weight: 10 +--- + +This guide explains how to configure cluster-autoscaler for automatic node scaling in Hetzner Cloud with Talos Linux. + +## Prerequisites + +- Hetzner Cloud account with API token +- `hcloud` CLI installed +- Existing Talos Kubernetes cluster +- [Networking Mesh]({{% ref "../networking-mesh" %}}) and [Local CCM]({{% ref "../local-ccm" %}}) configured + +## Step 1: Create Talos Image in Hetzner Cloud + +Hetzner doesn't support direct image uploads, so we need to create a snapshot via a temporary server. + +### 1.1 Generate Schematic ID + +Create a schematic at [factory.talos.dev](https://factory.talos.dev) with required extensions: + +```bash +curl -s -X POST https://factory.talos.dev/schematics \ + -H "Content-Type: application/json" \ + -d '{ + "customization": { + "systemExtensions": { + "officialExtensions": [ + "siderolabs/qemu-guest-agent", + "siderolabs/amd-ucode", + "siderolabs/amdgpu-firmware", + "siderolabs/bnx2-bnx2x", + "siderolabs/drbd", + "siderolabs/i915-ucode", + "siderolabs/intel-ice-firmware", + "siderolabs/intel-ucode", + "siderolabs/qlogic-firmware", + "siderolabs/zfs" + ] + } + } + }' +``` + +Save the returned `id` as `SCHEMATIC_ID`. + +{{% alert title="Note" color="info" %}} +`siderolabs/qemu-guest-agent` is required for Hetzner Cloud. Add other extensions +(zfs, drbd, etc.) as needed for your workloads. +{{% /alert %}} + +### 1.2 Configure hcloud CLI + +```bash +export HCLOUD_TOKEN="" +``` + +### 1.3 Create temporary server in rescue mode + +```bash +# Create server (without starting) +hcloud server create \ + --name talos-image-builder \ + --type cpx22 \ + --image ubuntu-24.04 \ + --location fsn1 \ + --ssh-key \ + --start-after-create=false + +# Enable rescue mode and start +hcloud server enable-rescue --type linux64 --ssh-key talos-image-builder +hcloud server poweron talos-image-builder +``` + +### 1.4 Write Talos image to disk + +```bash +# Get server IP +SERVER_IP=$(hcloud server ip talos-image-builder) + +# SSH into rescue mode and write image +ssh root@$SERVER_IP + +# Inside rescue mode: +wget -O- "https://factory.talos.dev/image/${SCHEMATIC_ID}//hcloud-amd64.raw.xz" \ + | xz -d \ + | dd of=/dev/sda bs=4M status=progress +sync +exit +``` + +### 1.5 Create snapshot and cleanup + +```bash +# Power off and create snapshot +hcloud server poweroff talos-image-builder +hcloud server create-image --type snapshot --description "Talos " talos-image-builder + +# Get snapshot ID (save this for later) +hcloud image list --type snapshot + +# Delete temporary server +hcloud server delete talos-image-builder +``` + +## Step 2: Create Hetzner vSwitch (Optional but Recommended) + +Create a private network for communication between nodes: + +```bash +# Create network +hcloud network create --name cozystack-vswitch --ip-range 10.100.0.0/16 + +# Add subnet for your region (eu-central covers FSN1, NBG1) +hcloud network add-subnet cozystack-vswitch \ + --type cloud \ + --network-zone eu-central \ + --ip-range 10.100.0.0/24 +``` + +## Step 3: Create Talos Machine Config + +From your cluster repository, generate a worker config file: + +```bash +talm template -t templates/worker.yaml --offline --full > nodes/hetzner.yaml +``` + +Then edit `nodes/hetzner.yaml` for Hetzner workers: + +1. Add Hetzner location metadata (see [Networking Mesh]({{% ref "../networking-mesh" %}})): + ```yaml + machine: + nodeAnnotations: + kilo.squat.ai/location: hetzner-cloud + kilo.squat.ai/persistent-keepalive: "20" + nodeLabels: + topology.kubernetes.io/zone: hetzner-cloud + ``` +2. Set public Kubernetes API endpoint: + Change `cluster.controlPlane.endpoint` to the **public** API server address (for example `https://:6443`). You can find this address in your kubeconfig or publish it via ingress. +3. Remove discovered installer/network sections: + Delete `machine.install` and `machine.network` sections from this file. +4. Set external cloud provider for kubelet (see [Local CCM]({{% ref "../local-ccm" %}})): + ```yaml + machine: + kubelet: + extraArgs: + cloud-provider: external + ``` +5. Fix node IP subnet detection: + Set `machine.kubelet.nodeIP.validSubnets` to your vSwitch subnet (for example `10.100.0.0/24`). +6. (Optional) Add registry mirrors to avoid Docker Hub rate limiting: + ```yaml + machine: + registries: + mirrors: + docker.io: + endpoints: + - https://mirror.gcr.io + ``` + +Result should include at least: + +```yaml +machine: + nodeAnnotations: + kilo.squat.ai/location: hetzner-cloud + kilo.squat.ai/persistent-keepalive: "20" + nodeLabels: + topology.kubernetes.io/zone: hetzner-cloud + kubelet: + nodeIP: + validSubnets: + - 10.100.0.0/24 # replace with your vSwitch subnet + extraArgs: + cloud-provider: external + registries: + mirrors: + docker.io: + endpoints: + - https://mirror.gcr.io +cluster: + controlPlane: + endpoint: https://:6443 +``` + +All other settings (cluster tokens, CA, extensions, etc.) remain the same as the generated template. + +## Step 4: Create Kubernetes Secrets + +### 4.1 Create secret with Hetzner API token + +```bash +kubectl -n cozy-cluster-autoscaler-hetzner create secret generic hetzner-credentials \ + --from-literal=token= +``` + +### 4.2 Create secret with Talos machine config + +The machine config must be base64-encoded: + +```bash +# Encode your worker.yaml (single line base64) +base64 -w 0 -i worker.yaml -o worker.b64 + +# Create secret +kubectl -n cozy-cluster-autoscaler-hetzner create secret generic talos-config \ + --from-file=cloud-init=worker.b64 +``` + +## Step 5: Deploy Cluster Autoscaler + +Create the Package resource: + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: Package +metadata: + name: cozystack.cluster-autoscaler-hetzner +spec: + variant: default + components: + cluster-autoscaler-hetzner: + values: + cluster-autoscaler: + autoscalingGroups: + - name: workers-fsn1 + minSize: 0 + maxSize: 10 + instanceType: cpx22 + region: FSN1 + extraEnv: + HCLOUD_IMAGE: "" + HCLOUD_SSH_KEY: "" + HCLOUD_NETWORK: "cozystack-vswitch" + HCLOUD_PUBLIC_IPV4: "true" + HCLOUD_PUBLIC_IPV6: "false" + extraEnvSecrets: + HCLOUD_TOKEN: + name: hetzner-credentials + key: token + HCLOUD_CLOUD_INIT: + name: talos-config + key: cloud-init +``` + +Apply: +```bash +kubectl apply -f package.yaml +``` + +## Step 6: Test Autoscaling + +Create a deployment with pod anti-affinity to force scale-up: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: test-autoscaler +spec: + replicas: 5 + selector: + matchLabels: + app: test-autoscaler + template: + metadata: + labels: + app: test-autoscaler + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app: test-autoscaler + topologyKey: kubernetes.io/hostname + containers: + - name: nginx + image: nginx + resources: + requests: + cpu: "100m" + memory: "128Mi" +``` + +If you have fewer nodes than replicas, the autoscaler will create new Hetzner servers. + +## Step 7: Verify + +```bash +# Check autoscaler logs +kubectl -n cozy-cluster-autoscaler-hetzner logs \ + deployment/cluster-autoscaler-hetzner-hetzner-cluster-autoscaler -f + +# Check nodes +kubectl get nodes -o wide + +# Verify node labels and internal IP +kubectl get node --show-labels +``` + +Expected result for autoscaled nodes: +- Internal IP from vSwitch range (e.g., 10.100.0.2) +- Label `kilo.squat.ai/location=hetzner-cloud` + +## Configuration Reference + +### Environment Variables + +| Variable | Description | Required | +|----------|-------------|----------| +| `HCLOUD_TOKEN` | Hetzner API token | Yes | +| `HCLOUD_IMAGE` | Talos snapshot ID | Yes | +| `HCLOUD_CLOUD_INIT` | Base64-encoded machine config | Yes | +| `HCLOUD_NETWORK` | vSwitch network name/ID | No | +| `HCLOUD_SSH_KEY` | SSH key name/ID | No | +| `HCLOUD_FIREWALL` | Firewall name/ID | No | +| `HCLOUD_PUBLIC_IPV4` | Assign public IPv4 | No (default: true) | +| `HCLOUD_PUBLIC_IPV6` | Assign public IPv6 | No (default: false) | + +### Hetzner Server Types + +| Type | vCPU | RAM | Good for | +|------|------|-----|----------| +| cpx22 | 2 | 4GB | Small workloads | +| cpx32 | 4 | 8GB | General purpose | +| cpx42 | 8 | 16GB | Medium workloads | +| cpx52 | 16 | 32GB | Large workloads | +| ccx13 | 2 dedicated | 8GB | CPU-intensive | +| ccx23 | 4 dedicated | 16GB | CPU-intensive | +| ccx33 | 8 dedicated | 32GB | CPU-intensive | +| cax11 | 2 ARM | 4GB | ARM workloads | +| cax21 | 4 ARM | 8GB | ARM workloads | + +{{% alert title="Note" color="info" %}} +Some older server types (cpx11, cpx21, etc.) may be unavailable in certain regions. +{{% /alert %}} + +### Hetzner Regions + +| Code | Location | +|------|----------| +| FSN1 | Falkenstein, Germany | +| NBG1 | Nuremberg, Germany | +| HEL1 | Helsinki, Finland | +| ASH | Ashburn, USA | +| HIL | Hillsboro, USA | + +## Troubleshooting + +### Connecting to remote workers for diagnostics + +Talos does not allow opening a dashboard directly to worker nodes. Use `talm dashboard` +to connect through the control plane: + +```bash +talm dashboard -f nodes/.yaml -n +``` + +Where `.yaml` is your control plane node config and `` is +the Kubernetes internal IP of the remote worker. + +### Nodes not joining cluster + +1. Check VNC console via Hetzner Cloud Console or: + ```bash + hcloud server request-console + ``` +2. Common errors: + - **"unknown keys found during decoding"**: Check Talos config format. `nodeLabels` goes under `machine`, `nodeIP` goes under `machine.kubelet` + - **"kubelet image is not valid"**: Kubernetes version mismatch. Use kubelet version compatible with your Talos version + - **"failed to load config"**: Machine config syntax error + +### Nodes have wrong Internal IP + +Ensure `machine.kubelet.nodeIP.validSubnets` is set to your vSwitch subnet: +```yaml +machine: + kubelet: + nodeIP: + validSubnets: + - 10.100.0.0/24 +``` + +### Scale-up not triggered + +1. Check autoscaler logs for errors +2. Verify RBAC permissions (leases access required) +3. Check if pods are actually pending: + ```bash + kubectl get pods --field-selector=status.phase=Pending + ``` + +### Registry rate limiting (403 errors) + +Add registry mirrors to Talos config: +```yaml +machine: + registries: + mirrors: + docker.io: + endpoints: + - https://mirror.gcr.io + registry.k8s.io: + endpoints: + - https://registry.k8s.io +``` + +### Scale-down not working + +The autoscaler caches node information for up to 30 minutes. Wait or restart autoscaler: +```bash +kubectl -n cozy-cluster-autoscaler-hetzner rollout restart \ + deployment cluster-autoscaler-hetzner-hetzner-cluster-autoscaler +``` diff --git a/content/en/docs/v1/operations/multi-location/local-ccm.md b/content/en/docs/v1/operations/multi-location/local-ccm.md new file mode 100644 index 00000000..e633ee6b --- /dev/null +++ b/content/en/docs/v1/operations/multi-location/local-ccm.md @@ -0,0 +1,39 @@ +--- +title: "Local Cloud Controller Manager" +linkTitle: "Local CCM" +description: "Node IP detection and lifecycle management for multi-location clusters." +weight: 15 +--- + +The `local-ccm` package provides a lightweight cloud controller manager for self-managed clusters. +It handles node IP detection and node lifecycle without requiring an external cloud provider. + +## What it does + +- **External IP detection**: Detects each node's external IP via `ip route get` (default target: `8.8.8.8`) +- **Node initialization**: Removes the `node.cloudprovider.kubernetes.io/uninitialized` taint so pods can be scheduled +- **Node lifecycle controller** (optional): Monitors NotReady nodes via ICMP ping and removes them after a configurable timeout + +## Install + +```bash +cozypkg add cozystack.local-ccm +``` + +## Talos machine config + +All nodes in the cluster (including control plane) must have `cloud-provider: external` set +so that kubelet defers node initialization to the cloud controller manager: + +```yaml +machine: + kubelet: + extraArgs: + cloud-provider: external +``` + +{{% alert title="Important" color="warning" %}} +The `cloud-provider: external` setting must be present on **all** nodes in the cluster, +including control plane nodes. Without it, the cluster-autoscaler cannot match Kubernetes +nodes to cloud provider instances (e.g. Azure VMSS). +{{% /alert %}} diff --git a/content/en/docs/v1/operations/multi-location/networking-mesh.md b/content/en/docs/v1/operations/multi-location/networking-mesh.md new file mode 100644 index 00000000..58850bfa --- /dev/null +++ b/content/en/docs/v1/operations/multi-location/networking-mesh.md @@ -0,0 +1,105 @@ +--- +title: "Networking Mesh" +linkTitle: "Networking Mesh" +description: "Configure Kilo WireGuard mesh with Cilium for multi-location cluster connectivity." +weight: 10 +--- + +Kilo creates a WireGuard mesh between cluster locations. When running with Cilium, it uses +IPIP encapsulation routed through Cilium's VxLAN overlay so that traffic between locations +works even when the cloud network blocks raw IPIP (protocol 4) packets. + +{{% alert title="Compatibility" color="warning" %}} +Multi-location support has been tested with the **Cilium** networking variant only. +The **KubeOVN+Cilium** variant has not been tested yet. +{{% /alert %}} + +## Install Kilo + +```bash +cozypkg add cozystack.kilo +``` + +When prompted, select the **cilium** variant. This deploys kilo with `--compatibility=cilium`, +enabling Cilium-aware IPIP encapsulation. + +## Configure Cilium + +### Disable host firewall + +Cilium host firewall drops IPIP (protocol 4) traffic because the protocol is not in +Cilium's network policy API +(see [cilium#44386](https://github.com/cilium/cilium/issues/44386)). +Disable it: + +```bash +kubectl patch package cozystack.networking --type merge -p ' +spec: + components: + cilium: + values: + cilium: + hostFirewall: + enabled: false +' +``` + +## How it works + +1. Kilo runs in `--local=false` mode -- it does not manage routes within a location (Cilium handles that) +2. Kilo creates a WireGuard tunnel (`kilo0`) between location leaders +3. Non-leader nodes in each location reach remote locations through IPIP encapsulation to their location leader, routed via Cilium's VxLAN overlay +4. The leader decapsulates IPIP and forwards traffic through the WireGuard tunnel + +## Talos machine config for cloud nodes + +Cloud worker nodes must include Kilo annotations in their Talos machine config: + +```yaml +machine: + nodeAnnotations: + kilo.squat.ai/location: + kilo.squat.ai/persistent-keepalive: "20" + nodeLabels: + topology.kubernetes.io/zone: +``` + +{{% alert title="Note" color="info" %}} +Kilo reads `kilo.squat.ai/location` from **node annotations**, not labels. The +`persistent-keepalive` annotation is critical for cloud nodes behind NAT -- it enables +WireGuard NAT traversal, allowing Kilo to discover the real public endpoint automatically. +{{% /alert %}} + +## Allowed location IPs + +By default, Kilo only routes pod and service CIDRs through the WireGuard mesh. If nodes in a +location use a private subnet that other locations need to reach (e.g. for kubelet communication +or NodePort access), annotate the location leader node with `kilo.squat.ai/allowed-location-ips`: + +```yaml +machine: + nodeAnnotations: + kilo.squat.ai/allowed-location-ips: 192.168.102.0/24,192.168.103.0/24 +``` + +This tells Kilo to include the specified CIDRs in the WireGuard allowed IPs for that location, +making those subnets routable through the tunnel from all other locations. + +{{% alert title="Note" color="info" %}} +Set this annotation on the **location leader** node (the node elected by Kilo to terminate +the WireGuard tunnel for a given location). The annotation accepts a comma-separated list of +CIDRs. Typically you would list all node subnets used in that cloud location. +{{% /alert %}} + +## Troubleshooting + +### WireGuard tunnel not established +- Verify the node has `kilo.squat.ai/persistent-keepalive: "20"` annotation +- Verify the node has `kilo.squat.ai/location` annotation (not just as a label) +- Check that the cloud firewall allows inbound UDP 51820 +- Inspect kilo logs: `kubectl logs -n cozy-kilo ` +- Repeating "WireGuard configurations are different" messages every 30 seconds indicate a missing `persistent-keepalive` annotation + +### Non-leader nodes unreachable (kubectl logs/exec timeout) +- Verify IP forwarding is enabled on the cloud network interfaces (required for the Kilo leader to forward traffic) +- Check kilo pod logs for `failed to create tunnel interface` errors