commit 3d3ddc98e960b8b969333e073c176bfd2daaca57 Author: Philipp Traber Date: Sun Sep 27 12:16:27 2026 +0200 initial commit: cirrus edge (Talos + sish) - talos/: generated base config (gitignored) + patches for control-plane scheduling, unprivileged ports and the ingress firewall - kubernetes/: sish base and cirrus-dev overlay, applied with kubectl - READMEs incl. production rollout plan Co-Authored-By: Claude Opus 5.5 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..4fd5b20 --- /dev/null +++ b/.gitignore @@ -0,0 +1,10 @@ +# Plaintext secrets: never commit, back them up outside this folder +.secrets/ + +# Talos generated configs contain cluster PKI and admin credentials (any folder, +# e.g. a new edge's config generated next to talos/) +controlplane.yaml +worker.yaml +talosconfig +secrets.yaml +kubeconfig diff --git a/README.md b/README.md new file mode 100644 index 0000000..1c0dd97 --- /dev/null +++ b/README.md @@ -0,0 +1,74 @@ +# cirrus + +Self-hosted edge: a Talos node running only [sish](https://github.com/antoniomika/sish). Clusters +without inbound ports (e.g. `cumulus`) open outbound SSH tunnels to it with `sish-client`, and the +edge relays public traffic back through them: + +``` +client ──▶ edge :22/:80/:443/:200xx (sish) ══ssh══▶ sish-client ──▶ envoy gateway ──▶ app +``` + +- :443 is routed by SNI without decrypting (TLS passthrough), optionally with a PROXY v2 header. +- :80 is routed by `Host` header, raw TCP ports (e.g. :22 for Gitea SSH) by port. + +## Layout + +``` +talos/ node config: generated base + patches (sysctl, scheduling, firewall) → talos/README.md +kubernetes/ sish deployment, kustomize base + per-edge overlay → kubernetes/README.md +.secrets/ connector private keys (gitignored) +``` + +Everything is applied by hand (`talosctl`, `kubectl apply -k`). Secrets never leave the +gitignored files (`talos/controlplane.yaml`, `talos/talosconfig`, `**/.secrets/`). + +## Environments + +| | Edge | Domains | +|---|---|---| +| `cirrus-dev` | `10.20.5.130` (LAN), Talos v1.14.1, Kubernetes v1.37.0 | `.test` / `.sto` via local DNS | + +Currently served through the dev edge from `cumulus`: Gitea SSH on :22, `http://cirrus.sto` (hello). + +## Production rollout plan + +The dev edge is verified end to end: firewall, port range, tunnels, and the Talos config +reproduces exactly from the files here. What is still missing for production: + +**Before the rollout** +1. **Backups.** `talos/controlplane.yaml`, `talos/talosconfig`, the edge host key and the + connector private keys exist only in this folder. Store them in a password manager (and put + the folder under git, secrets stay gitignored). +2. **Domains and DNS.** Pick the production domains; create DNS records for them (wildcards + where needed) pointing to the VPS IPv4/IPv6. +3. **HTTPS on the cluster side.** `cumulus` only has the plain connector (SSH, HTTP). For + TLS passthrough it needs a second connector (SNI, PROXY v2), an Envoy HTTPS listener that + accepts the PROXY header only from sish-client, and certificates (cert-manager with DNS-01, or + HTTP-01 over the :80 route). +4. **Remove test access.** Leave `connector-hello.pub` (local test stack) out of the + production overlay; give each production connector its own key. + +**Rollout** +1. VPS: boot the Talos image (same factory schematic as dev) and check the provider's disk name + and network (DHCP vs static, IPv6). +2. Generate a new base config into its own folder (`talos/README.md`, "New node") and apply it + with the existing patches plus one that disables the discovery service (single node, no + external dependency needed). +3. Bootstrap, fetch the kubeconfig. Verify against the firewall table: only the listed ports + answer from outside. +4. Kubernetes: new overlay `kubernetes/cirrus-prod` (copy of `cirrus-dev`) with the production + `SISH_DOMAIN`/`SISH_BIND_HOSTS`, production connector keys and a new host key. Set + `dnsPolicy: Default` for sish so the tunnel does not depend on CoreDNS. `diff`, then `apply -k`. +5. Cluster side (`cumulus`): sish-client deployment(s) for the production edge (edge IP, new host + key, production routes), matching Gateway listeners/routes and network policies. Keep the dev + connectors until production is verified. +6. Verify from outside: `ssh-keyscan` for SSH routes, `curl` for HTTP/HTTPS routes, the backend + sees the real client IP, closed ports stay closed. +7. Move existing services (e.g. from the Cloudflare tunnel) one hostname at a time by switching + their DNS records. + +**After the rollout** +- External uptime check on the edge (sish :2222 and one route per protocol): the edge is a + single point of failure for everything behind it. +- Updates by hand: `talosctl upgrade` / `upgrade-k8s`, the sish image tag, sish-client tags in + `cumulus`. The `talosconfig` admin certificate expires after one year. diff --git a/kubernetes/README.md b/kubernetes/README.md new file mode 100644 index 0000000..4cca951 --- /dev/null +++ b/kubernetes/README.md @@ -0,0 +1,58 @@ +# Kubernetes: sish on the edge + +Plain kustomize, applied by hand. `base/sish` is the generic deployment, each edge gets an +overlay (`cirrus-dev/`) with its domain, allowed hostnames, connector keys and host key. + +```sh +kubectl --context admin@cirrus_dev diff -k kubernetes/cirrus-dev # review first +kubectl --context admin@cirrus_dev apply -k kubernetes/cirrus-dev +kubectl --context admin@cirrus_dev -n sish logs deploy/sish -f +``` + +Config changes create a new ConfigMap/Secret name (kustomize hash), which rolls the pod. +Rollouts use `Recreate` (host ports cannot be shared), so connectors drop for a few seconds +and reconnect on their own. + +## sish + +- `hostNetwork`, non-root (uid 65534), no capabilities, read-only root filesystem. Binding + :22/:80/:443 relies on the Talos sysctl in `talos/patches/unprivileged-ports.yaml`. +- `config.yml`: SNI passthrough on :443 (sish never terminates TLS), HTTP by `Host` on :80, + raw TCP forwards, public-key auth only, no web consoles. +- `port-bind-range` (ports connectors may claim) must match the firewall rule in + `talos/patches/firewall.yaml`: 22, 443 and 20000-20099 for raw TCP forwards. +- Several connectors claiming the same host/port are load-balanced round-robin. + +## Overlay `cirrus-dev` + +| | | +|---|---| +| `SISH_DOMAIN` | Fallback domain sish prints for forwards | +| `SISH_BIND_HOSTS` | Parent domains connectors may claim hostnames under (exact match on everything after the first label) | +| `pubkeys/*.pub` | Authorized connector public keys, one file per connector | +| `.secrets/ssh_host_ed25519_key` | Edge SSH host key (gitignored). Connectors pin its public half (`SISH_HOST_KEY`) | + +Add a connector: put its public key into `pubkeys/`, list it under `sish-pubkeys` in +`kustomization.yaml`, then diff and apply. Remove a connector the same way; its sessions are +cut when the pod restarts. + +New host key (e.g. for a new edge): `ssh-keygen -t ed25519 -N '' -C sish-host@ -f +kubernetes//.secrets/ssh_host_ed25519_key`, then update `SISH_HOST_KEY` in every +connector. + +Connectors run `registry.traberph.de/public/sish-client` (see its repo README); the +`cumulus` cluster runs one in `infra/base/networking/sish-client`. + +## sish gotchas + +- `port-bind-range` defaults to `0,1024-65535`, which rejects 22 and 443. +- A raw TCP forward must bind `0.0.0.0:`. With a hostname sish creates a TCP *alias*, + reachable only through sish itself, not as a public port. +- PROXY protocol is opt-in per connector (`proxy-protocol=…`); the server only fixes the version. +- `idle-connection-timeout` defaults to 5s; raised to 1h. +- `service-console-max-content-length` defaults to -1, which buffers every HTTP body in memory + (even with consoles off): a large upload OOM-kills sish. Set to 0 to stream. +- A name outside `bind-hosts` is not rejected: sish silently binds `.` instead. + Check the `HTTP:`/`TLS:` line the edge prints. +- HTTP (:80) and SNI (:443) forwards need separate connector sessions. +- `Can't read file ..data` log lines are harmless (Kubernetes volume symlinks). diff --git a/kubernetes/base/sish/config.yml b/kubernetes/base/sish/config.yml new file mode 100644 index 0000000..56a12f5 --- /dev/null +++ b/kubernetes/base/sish/config.yml @@ -0,0 +1,52 @@ +# sish configuration (keys mirror the CLI flags, see `sish --help`). +# Cluster-specific values (domain, bind-hosts) are injected as SISH_* env vars +# from the `sish-env` ConfigMap in each overlay. Env takes precedence over this file. + +# Listeners +ssh-address: ":2222" +http-address: ":80" +https: false # sish never terminates TLS; :443 is an SNI passthrough listener + +# SNI passthrough + multiple connectors per hostname +sni-proxy: true +sni-load-balancer: true +tcp-load-balancer: true +http-load-balancer: true + +# Connectors get exactly what they ask for, or the bind fails +bind-random-ports: false +bind-random-subdomains: false +bind-random-aliases: false +force-requested-subdomains: true +force-requested-ports: true +bind-wildcards: true +# Ports connectors may claim. Must match the sish-public rule in +# talos/patches/firewall.yaml, otherwise a claimed port is silently unreachable. +# 22 gitea ssh, 443 SNI, 20000-20099 reserved for raw tcp forwards. +# Ports below 80 also need talos/patches/unprivileged-ports.yaml. +port-bind-range: "22,443,20000-20099" + +# PROXY header version for connectors that request it (sish-client default: v2) +proxy-protocol: true +proxy-protocol-version: "2" + +# Default is 5s, which kills idle websockets/SSE/slow uploads +idle-connection-timeout: 1h + +# Auth: public keys only +authentication: true +authentication-keys-directory: /pubkeys +private-keys-directory: /keys + +# No web UI / consoles +redirect-root: false +admin-console: false +service-console: false +load-templates: false +# Default -1 makes the HTTP muxer io.ReadAll() every request/response body into memory +# (for the console), even with consoles disabled: large uploads OOM-kill sish. +# 0 = never buffer, stream bodies through. +service-console-max-content-length: 0 + +log-to-stdout: true +log-to-file: false diff --git a/kubernetes/base/sish/deployment.yaml b/kubernetes/base/sish/deployment.yaml new file mode 100644 index 0000000..94160c7 --- /dev/null +++ b/kubernetes/base/sish/deployment.yaml @@ -0,0 +1,94 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: sish + labels: + app.kubernetes.io/name: sish +spec: + replicas: 1 + # Host ports cannot be shared, so the old pod must be gone before the new one starts. + strategy: + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: sish + template: + metadata: + labels: + app.kubernetes.io/name: sish + spec: + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + enableServiceLinks: false + automountServiceAccountToken: false + # Binding :22/:80/:443 as non-root relies on the node sysctl + # net.ipv4.ip_unprivileged_port_start=22 (talos/patches/unprivileged-ports.yaml). + securityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + fsGroup: 65534 + seccompProfile: + type: RuntimeDefault + containers: + - name: sish + image: docker.io/antoniomika/sish:v2.23.0 + args: + - --config=/config/config.yml + envFrom: + - configMapRef: + name: sish-env + optional: true + ports: + - name: ssh + containerPort: 2222 + - name: http + containerPort: 80 + - name: https + containerPort: 443 + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 20m + memory: 32Mi + limits: + memory: 256Mi + readinessProbe: + tcpSocket: + port: ssh + periodSeconds: 10 + livenessProbe: + tcpSocket: + port: ssh + initialDelaySeconds: 10 + periodSeconds: 20 + volumeMounts: + - name: config + mountPath: /config + readOnly: true + - name: hostkey + mountPath: /keys + readOnly: true + - name: pubkeys + mountPath: /pubkeys + readOnly: true + - name: tmp + mountPath: /tmp + volumes: + - name: config + configMap: + name: sish-config + - name: hostkey + secret: + secretName: sish-hostkey + defaultMode: 0440 + - name: pubkeys + configMap: + name: sish-pubkeys + - name: tmp + emptyDir: + sizeLimit: 16Mi diff --git a/kubernetes/base/sish/kustomization.yaml b/kubernetes/base/sish/kustomization.yaml new file mode 100644 index 0000000..80a05d7 --- /dev/null +++ b/kubernetes/base/sish/kustomization.yaml @@ -0,0 +1,14 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: sish +resources: + - namespace.yaml + - deployment.yaml +configMapGenerator: + - name: sish-config + files: + - config.yml +# Provided by each overlay: +# ConfigMap sish-env (SISH_DOMAIN, SISH_BIND_HOSTS) +# ConfigMap sish-pubkeys (authorized connector public keys) +# Secret sish-hostkey (pinned SSH host key) diff --git a/kubernetes/base/sish/namespace.yaml b/kubernetes/base/sish/namespace.yaml new file mode 100644 index 0000000..2931e57 --- /dev/null +++ b/kubernetes/base/sish/namespace.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: sish + labels: + # hostNetwork is only permitted by the privileged profile. + # The pod itself still runs non-root with all capabilities dropped. + pod-security.kubernetes.io/enforce: privileged + pod-security.kubernetes.io/audit: baseline diff --git a/kubernetes/cirrus-dev/kustomization.yaml b/kubernetes/cirrus-dev/kustomization.yaml new file mode 100644 index 0000000..b898647 --- /dev/null +++ b/kubernetes/cirrus-dev/kustomization.yaml @@ -0,0 +1,23 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: sish +resources: + - ../base/sish +configMapGenerator: + - name: sish-env + literals: + # .test is a reserved TLD: fine for dev, replace with real domains on the edge. + - SISH_DOMAIN=tunnel.cirrus.test + # Parents a connector may bind under (exact match on everything after the first label). + # "sto" allows cirrus.sto itself (and any .sto); "cirrus.sto" allows .cirrus.sto + - SISH_BIND_HOSTS=cirrus.test,apps.cirrus.test,sto,cirrus.sto + - name: sish-pubkeys + files: + - pubkeys/connector-dev.pub + - pubkeys/connector-hello.pub +secretGenerator: + # Edge SSH host key (connectors pin its public half). Kept only locally in + # .secrets/ (gitignored), so back it up outside this folder. + - name: sish-hostkey + files: + - ssh_host_ed25519_key=.secrets/ssh_host_ed25519_key diff --git a/kubernetes/cirrus-dev/pubkeys/connector-dev.pub b/kubernetes/cirrus-dev/pubkeys/connector-dev.pub new file mode 100644 index 0000000..cebcb8b --- /dev/null +++ b/kubernetes/cirrus-dev/pubkeys/connector-dev.pub @@ -0,0 +1 @@ +ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOHcrMOuL1K6TPPprZi/W3N29SBRN23pAGmxp3SfQ6qr connector-dev diff --git a/kubernetes/cirrus-dev/pubkeys/connector-hello.pub b/kubernetes/cirrus-dev/pubkeys/connector-hello.pub new file mode 100644 index 0000000..b3dd916 --- /dev/null +++ b/kubernetes/cirrus-dev/pubkeys/connector-hello.pub @@ -0,0 +1 @@ +ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINZ1rT2t5rUS3tJesxzY7HogcBRCvzMT8HF+xNddV7e1 connector-hello diff --git a/talos/README.md b/talos/README.md new file mode 100644 index 0000000..6b1f97a --- /dev/null +++ b/talos/README.md @@ -0,0 +1,77 @@ +# Talos: cirrus edge node + +Single-node Talos control plane that runs only sish. Talos and Kubernetes are managed by hand +with `talosctl`/`kubectl` (no Flux). + +| File | | +|---|---| +| `controlplane.yaml` | Base config, **unmodified** output of `talosctl gen config` (gitignored, contains the cluster PKI) | +| `worker.yaml` | Generated worker config, unused on a single node (gitignored) | +| `talosconfig` | Admin client config for `talosctl` (gitignored) | +| `patches/*.yaml` | Every change to the base config | + +```sh +export TALOSCONFIG=talos/talosconfig # run from the repo root +N="-n 10.20.5.130 -e 10.20.5.130" +``` + +## Base config + patches + +The base file is never edited by hand; all changes live in `patches/`. The node's config is +therefore always `controlplane.yaml` + `patches/*.yaml`, which keeps changes reviewable and lets +a newly generated base (new node, new Talos defaults) get the same changes by reapplying the +patches. `talosctl patch mc` merges a patch into the node's live config; it does not touch the +local files. + +| Patch | Purpose | +|---|---| +| `control-plane-scheduling.yaml` | Drops the control-plane `NoSchedule` taint so sish can run on the only node | +| `unprivileged-ports.yaml` | `ip_unprivileged_port_start=22`: sish (non-root, hostNetwork) binds :22/:80/:443 | +| `firewall.yaml` | Ingress firewall, default block (see below) | + +Apply a single patch (dry run first; use `--mode try` for anything that can lock you out, it +reverts automatically unless re-applied): + +```sh +talosctl $N patch mc --patch @talos/patches/.yaml --mode no-reboot --dry-run +talosctl $N patch mc --patch @talos/patches/.yaml --mode no-reboot +``` + +Check that the node matches the files (expect `No changes.`): + +```sh +talosctl machineconfig patch talos/controlplane.yaml \ + $(for p in talos/patches/*.yaml; do printf -- '--patch @%s ' "$p"; done) | + talosctl $N apply-config --file /dev/stdin --dry-run +``` + +## Firewall + +Default action `block`. Loopback and replies to outgoing connections are always allowed. + +| Open | Source | | +|---|---|---| +| 22, 80, 443, 2222, 20000-20099 tcp | anyone | sish (2222 = connector SSH, rest = forwards) | +| 6443, 50000 tcp | anyone | Kubernetes API, Talos API (both client-cert authenticated) | +| 53 udp/tcp | pod network `10.244.0.0/16` | CoreDNS forwards to the Talos host DNS | + +Closed: flannel VXLAN 4789/udp, etcd 2379-2383, kubelet 10250, kube-proxy 10256, trustd 50001 +(open it to the node network only when a second node joins). + +The sish ports must match `port-bind-range` in `kubernetes/base/sish/config.yml`. To add a raw +TCP port outside 20000-20099, change both files together. + +## New node + +```sh +talosctl gen config https://:6443 --install-disk -o talos/ +talosctl machineconfig patch talos/controlplane.yaml \ + $(for p in talos/patches/*.yaml; do printf -- '--patch @%s ' "$p"; done) | + talosctl apply-config --insecure -n --file /dev/stdin +talosctl --talosconfig talos/talosconfig config endpoint +talosctl --talosconfig talos/talosconfig -n bootstrap +talosctl --talosconfig talos/talosconfig -n kubeconfig +``` + +Back up `controlplane.yaml` and `talosconfig` outside this folder: they are the only copy of +the cluster PKI and admin credentials. diff --git a/talos/patches/control-plane-scheduling.yaml b/talos/patches/control-plane-scheduling.yaml new file mode 100644 index 0000000..96cbeaa --- /dev/null +++ b/talos/patches/control-plane-scheduling.yaml @@ -0,0 +1,7 @@ +# Single node: let workloads (sish) run on the control plane by dropping the +# default control-plane NoSchedule taint. +apiVersion: v1alpha1 +kind: KubeNodeConfig +taints: + node-role.kubernetes.io/control-plane: + $patch: delete diff --git a/talos/patches/firewall.yaml b/talos/patches/firewall.yaml new file mode 100644 index 0000000..d94e27f --- /dev/null +++ b/talos/patches/firewall.yaml @@ -0,0 +1,61 @@ +# Ingress firewall: block everything that is not listed here. Loopback and +# replies to outgoing connections are always allowed by Talos. +# +# Public TCP ports served by sish must match port-bind-range in +# kubernetes/base/sish/config.yml (plus :80 HTTP and :2222 sish SSH). +# Closed on purpose: flannel VXLAN 4789/udp (single node, unauthenticated), +# etcd 2379-2383, kubelet 10250, kube-proxy 10256, trustd 50001 (only needed +# when other nodes join). +apiVersion: v1alpha1 +kind: NetworkDefaultActionConfig +ingress: block +--- +apiVersion: v1alpha1 +kind: NetworkRuleConfig +name: sish-public +portSelector: + ports: + - 22 # gitea ssh (raw tcp forward) + - 80 # sish http, routed by Host header + - 443 # sish tls, routed by SNI + - 2222 # sish ssh endpoint for connectors (public key auth) + - 20000-20099 # reserved for raw tcp forwards + protocol: tcp +ingress: + - subnet: 0.0.0.0/0 + - subnet: ::/0 +--- +# Talos API (apid) and Kubernetes API, both mTLS / client-cert authenticated +apiVersion: v1alpha1 +kind: NetworkRuleConfig +name: talos-and-kube-api +portSelector: + ports: + - 50000 + - 6443 + protocol: tcp +ingress: + - subnet: 0.0.0.0/0 + - subnet: ::/0 +--- +# CoreDNS forwards to the Talos host DNS (forwardKubeDNSToHost), which pods reach +# on the host, so the pod network needs DNS to the node +apiVersion: v1alpha1 +kind: NetworkRuleConfig +name: pod-dns +portSelector: + ports: + - 53 + protocol: udp +ingress: + - subnet: 10.244.0.0/16 +--- +apiVersion: v1alpha1 +kind: NetworkRuleConfig +name: pod-dns-tcp +portSelector: + ports: + - 53 + protocol: tcp +ingress: + - subnet: 10.244.0.0/16 diff --git a/talos/patches/unprivileged-ports.yaml b/talos/patches/unprivileged-ports.yaml new file mode 100644 index 0000000..a7ec281 --- /dev/null +++ b/talos/patches/unprivileged-ports.yaml @@ -0,0 +1,6 @@ +# Lets non-root processes bind ports >= 22, so sish (hostNetwork, uid 65534, +# no capabilities) can listen on :22 (Gitea SSH), :80 and :443. +# The edge runs nothing else that could grab 22-79. +machine: + sysctls: + net.ipv4.ip_unprivileged_port_start: "22"