initial commit: cirrus edge (Talos + sish)

- talos/: generated base config (gitignored) + patches for control-plane scheduling, unprivileged ports and the ingress firewall - kubernetes/: sish base and cirrus-dev overlay, applied with kubectl - READMEs incl. production rollout plan Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 12:16:27 +02:00
commit 3d3ddc98e9
14 changed files with 487 additions and 0 deletions
@@ -0,0 +1,10 @@
+# Plaintext secrets: never commit, back them up outside this folder
+.secrets/
+
+# Talos generated configs contain cluster PKI and admin credentials (any folder,
+# e.g. a new edge's config generated next to talos/)
+controlplane.yaml
+worker.yaml
+talosconfig
+secrets.yaml
+kubeconfig
@@ -0,0 +1,74 @@
+# cirrus
+
+Self-hosted edge: a Talos node running only [sish](https://github.com/antoniomika/sish). Clusters
+without inbound ports (e.g. `cumulus`) open outbound SSH tunnels to it with `sish-client`, and the
+edge relays public traffic back through them:
+
+```
+client ──▶ edge :22/:80/:443/:200xx (sish) ══ssh══▶ sish-client ──▶ envoy gateway ──▶ app
+```
+
+- :443 is routed by SNI without decrypting (TLS passthrough), optionally with a PROXY v2 header.
+- :80 is routed by `Host` header, raw TCP ports (e.g. :22 for Gitea SSH) by port.
+
+## Layout
+
+```
+talos/        node config: generated base + patches (sysctl, scheduling, firewall)  → talos/README.md
+kubernetes/   sish deployment, kustomize base + per-edge overlay                   → kubernetes/README.md
+.secrets/     connector private keys (gitignored)
+```
+
+Everything is applied by hand (`talosctl`, `kubectl apply -k`). Secrets never leave the
+gitignored files (`talos/controlplane.yaml`, `talos/talosconfig`, `**/.secrets/`).
+
+## Environments
+
+| | Edge | Domains |
+|---|---|---|
+| `cirrus-dev` | `10.20.5.130` (LAN), Talos v1.14.1, Kubernetes v1.37.0 | `.test` / `.sto` via local DNS |
+
+Currently served through the dev edge from `cumulus`: Gitea SSH on :22, `http://cirrus.sto` (hello).
+
+## Production rollout plan
+
+The dev edge is verified end to end: firewall, port range, tunnels, and the Talos config
+reproduces exactly from the files here. What is still missing for production:
+
+**Before the rollout**
+1. **Backups.** `talos/controlplane.yaml`, `talos/talosconfig`, the edge host key and the
+   connector private keys exist only in this folder. Store them in a password manager (and put
+   the folder under git, secrets stay gitignored).
+2. **Domains and DNS.** Pick the production domains; create DNS records for them (wildcards
+   where needed) pointing to the VPS IPv4/IPv6.
+3. **HTTPS on the cluster side.** `cumulus` only has the plain connector (SSH, HTTP). For
+   TLS passthrough it needs a second connector (SNI, PROXY v2), an Envoy HTTPS listener that
+   accepts the PROXY header only from sish-client, and certificates (cert-manager with DNS-01, or
+   HTTP-01 over the :80 route).
+4. **Remove test access.** Leave `connector-hello.pub` (local test stack) out of the
+   production overlay; give each production connector its own key.
+
+**Rollout**
+1. VPS: boot the Talos image (same factory schematic as dev) and check the provider's disk name
+   and network (DHCP vs static, IPv6).
+2. Generate a new base config into its own folder (`talos/README.md`, "New node") and apply it
+   with the existing patches plus one that disables the discovery service (single node, no
+   external dependency needed).
+3. Bootstrap, fetch the kubeconfig. Verify against the firewall table: only the listed ports
+   answer from outside.
+4. Kubernetes: new overlay `kubernetes/cirrus-prod` (copy of `cirrus-dev`) with the production
+   `SISH_DOMAIN`/`SISH_BIND_HOSTS`, production connector keys and a new host key. Set
+   `dnsPolicy: Default` for sish so the tunnel does not depend on CoreDNS. `diff`, then `apply -k`.
+5. Cluster side (`cumulus`): sish-client deployment(s) for the production edge (edge IP, new host
+   key, production routes), matching Gateway listeners/routes and network policies. Keep the dev
+   connectors until production is verified.
+6. Verify from outside: `ssh-keyscan` for SSH routes, `curl` for HTTP/HTTPS routes, the backend
+   sees the real client IP, closed ports stay closed.
+7. Move existing services (e.g. from the Cloudflare tunnel) one hostname at a time by switching
+   their DNS records.
+
+**After the rollout**
+- External uptime check on the edge (sish :2222 and one route per protocol): the edge is a
+  single point of failure for everything behind it.
+- Updates by hand: `talosctl upgrade` / `upgrade-k8s`, the sish image tag, sish-client tags in
+  `cumulus`. The `talosconfig` admin certificate expires after one year.
@@ -0,0 +1,58 @@
+# Kubernetes: sish on the edge
+
+Plain kustomize, applied by hand. `base/sish` is the generic deployment, each edge gets an
+overlay (`cirrus-dev/`) with its domain, allowed hostnames, connector keys and host key.
+
+```sh
+kubectl --context admin@cirrus_dev diff -k kubernetes/cirrus-dev    # review first
+kubectl --context admin@cirrus_dev apply -k kubernetes/cirrus-dev
+kubectl --context admin@cirrus_dev -n sish logs deploy/sish -f
+```
+
+Config changes create a new ConfigMap/Secret name (kustomize hash), which rolls the pod.
+Rollouts use `Recreate` (host ports cannot be shared), so connectors drop for a few seconds
+and reconnect on their own.
+
+## sish
+
+- `hostNetwork`, non-root (uid 65534), no capabilities, read-only root filesystem. Binding
+  :22/:80/:443 relies on the Talos sysctl in `talos/patches/unprivileged-ports.yaml`.
+- `config.yml`: SNI passthrough on :443 (sish never terminates TLS), HTTP by `Host` on :80,
+  raw TCP forwards, public-key auth only, no web consoles.
+- `port-bind-range` (ports connectors may claim) must match the firewall rule in
+  `talos/patches/firewall.yaml`: 22, 443 and 20000-20099 for raw TCP forwards.
+- Several connectors claiming the same host/port are load-balanced round-robin.
+
+## Overlay `cirrus-dev`
+
+| | |
+|---|---|
+| `SISH_DOMAIN` | Fallback domain sish prints for forwards |
+| `SISH_BIND_HOSTS` | Parent domains connectors may claim hostnames under (exact match on everything after the first label) |
+| `pubkeys/*.pub` | Authorized connector public keys, one file per connector |
+| `.secrets/ssh_host_ed25519_key` | Edge SSH host key (gitignored). Connectors pin its public half (`SISH_HOST_KEY`) |
+
+Add a connector: put its public key into `pubkeys/`, list it under `sish-pubkeys` in
+`kustomization.yaml`, then diff and apply. Remove a connector the same way; its sessions are
+cut when the pod restarts.
+
+New host key (e.g. for a new edge): `ssh-keygen -t ed25519 -N '' -C sish-host@<edge> -f
+kubernetes/<overlay>/.secrets/ssh_host_ed25519_key`, then update `SISH_HOST_KEY` in every
+connector.
+
+Connectors run `registry.traberph.de/public/sish-client` (see its repo README); the
+`cumulus` cluster runs one in `infra/base/networking/sish-client`.
+
+## sish gotchas
+
+- `port-bind-range` defaults to `0,1024-65535`, which rejects 22 and 443.
+- A raw TCP forward must bind `0.0.0.0:<port>`. With a hostname sish creates a TCP *alias*,
+  reachable only through sish itself, not as a public port.
+- PROXY protocol is opt-in per connector (`proxy-protocol=…`); the server only fixes the version.
+- `idle-connection-timeout` defaults to 5s; raised to 1h.
+- `service-console-max-content-length` defaults to -1, which buffers every HTTP body in memory
+  (even with consoles off): a large upload OOM-kills sish. Set to 0 to stream.
+- A name outside `bind-hosts` is not rejected: sish silently binds `<name>.<domain>` instead.
+  Check the `HTTP:`/`TLS:` line the edge prints.
+- HTTP (:80) and SNI (:443) forwards need separate connector sessions.
+- `Can't read file ..data` log lines are harmless (Kubernetes volume symlinks).
@@ -0,0 +1,52 @@
+# sish configuration (keys mirror the CLI flags, see `sish --help`).
+# Cluster-specific values (domain, bind-hosts) are injected as SISH_* env vars
+# from the `sish-env` ConfigMap in each overlay. Env takes precedence over this file.
+
+# Listeners
+ssh-address: ":2222"
+http-address: ":80"
+https: false                      # sish never terminates TLS; :443 is an SNI passthrough listener
+
+# SNI passthrough + multiple connectors per hostname
+sni-proxy: true
+sni-load-balancer: true
+tcp-load-balancer: true
+http-load-balancer: true
+
+# Connectors get exactly what they ask for, or the bind fails
+bind-random-ports: false
+bind-random-subdomains: false
+bind-random-aliases: false
+force-requested-subdomains: true
+force-requested-ports: true
+bind-wildcards: true
+# Ports connectors may claim. Must match the sish-public rule in
+# talos/patches/firewall.yaml, otherwise a claimed port is silently unreachable.
+# 22 gitea ssh, 443 SNI, 20000-20099 reserved for raw tcp forwards.
+# Ports below 80 also need talos/patches/unprivileged-ports.yaml.
+port-bind-range: "22,443,20000-20099"
+
+# PROXY header version for connectors that request it (sish-client default: v2)
+proxy-protocol: true
+proxy-protocol-version: "2"
+
+# Default is 5s, which kills idle websockets/SSE/slow uploads
+idle-connection-timeout: 1h
+
+# Auth: public keys only
+authentication: true
+authentication-keys-directory: /pubkeys
+private-keys-directory: /keys
+
+# No web UI / consoles
+redirect-root: false
+admin-console: false
+service-console: false
+load-templates: false
+# Default -1 makes the HTTP muxer io.ReadAll() every request/response body into memory
+# (for the console), even with consoles disabled: large uploads OOM-kill sish.
+# 0 = never buffer, stream bodies through.
+service-console-max-content-length: 0
+
+log-to-stdout: true
+log-to-file: false
@@ -0,0 +1,94 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: sish
+  labels:
+    app.kubernetes.io/name: sish
+spec:
+  replicas: 1
+  # Host ports cannot be shared, so the old pod must be gone before the new one starts.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: sish
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: sish
+    spec:
+      hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
+      enableServiceLinks: false
+      automountServiceAccountToken: false
+      # Binding :22/:80/:443 as non-root relies on the node sysctl
+      # net.ipv4.ip_unprivileged_port_start=22 (talos/patches/unprivileged-ports.yaml).
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        runAsGroup: 65534
+        fsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: sish
+          image: docker.io/antoniomika/sish:v2.23.0
+          args:
+            - --config=/config/config.yml
+          envFrom:
+            - configMapRef:
+                name: sish-env
+                optional: true
+          ports:
+            - name: ssh
+              containerPort: 2222
+            - name: http
+              containerPort: 80
+            - name: https
+              containerPort: 443
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          resources:
+            requests:
+              cpu: 20m
+              memory: 32Mi
+            limits:
+              memory: 256Mi
+          readinessProbe:
+            tcpSocket:
+              port: ssh
+            periodSeconds: 10
+          livenessProbe:
+            tcpSocket:
+              port: ssh
+            initialDelaySeconds: 10
+            periodSeconds: 20
+          volumeMounts:
+            - name: config
+              mountPath: /config
+              readOnly: true
+            - name: hostkey
+              mountPath: /keys
+              readOnly: true
+            - name: pubkeys
+              mountPath: /pubkeys
+              readOnly: true
+            - name: tmp
+              mountPath: /tmp
+      volumes:
+        - name: config
+          configMap:
+            name: sish-config
+        - name: hostkey
+          secret:
+            secretName: sish-hostkey
+            defaultMode: 0440
+        - name: pubkeys
+          configMap:
+            name: sish-pubkeys
+        - name: tmp
+          emptyDir:
+            sizeLimit: 16Mi
@@ -0,0 +1,14 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: sish
+resources:
+  - namespace.yaml
+  - deployment.yaml
+configMapGenerator:
+  - name: sish-config
+    files:
+      - config.yml
+# Provided by each overlay:
+#   ConfigMap sish-env     (SISH_DOMAIN, SISH_BIND_HOSTS)
+#   ConfigMap sish-pubkeys (authorized connector public keys)
+#   Secret    sish-hostkey (pinned SSH host key)
@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: sish
+  labels:
+    # hostNetwork is only permitted by the privileged profile.
+    # The pod itself still runs non-root with all capabilities dropped.
+    pod-security.kubernetes.io/enforce: privileged
+    pod-security.kubernetes.io/audit: baseline
@@ -0,0 +1,23 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: sish
+resources:
+  - ../base/sish
+configMapGenerator:
+  - name: sish-env
+    literals:
+      # .test is a reserved TLD: fine for dev, replace with real domains on the edge.
+      - SISH_DOMAIN=tunnel.cirrus.test
+      # Parents a connector may bind under (exact match on everything after the first label).
+      # "sto" allows cirrus.sto itself (and any <name>.sto); "cirrus.sto" allows <name>.cirrus.sto
+      - SISH_BIND_HOSTS=cirrus.test,apps.cirrus.test,sto,cirrus.sto
+  - name: sish-pubkeys
+    files:
+      - pubkeys/connector-dev.pub
+      - pubkeys/connector-hello.pub
+secretGenerator:
+  # Edge SSH host key (connectors pin its public half). Kept only locally in
+  # .secrets/ (gitignored), so back it up outside this folder.
+  - name: sish-hostkey
+    files:
+      - ssh_host_ed25519_key=.secrets/ssh_host_ed25519_key
@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOHcrMOuL1K6TPPprZi/W3N29SBRN23pAGmxp3SfQ6qr connector-dev
@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINZ1rT2t5rUS3tJesxzY7HogcBRCvzMT8HF+xNddV7e1 connector-hello
@@ -0,0 +1,77 @@
+# Talos: cirrus edge node
+
+Single-node Talos control plane that runs only sish. Talos and Kubernetes are managed by hand
+with `talosctl`/`kubectl` (no Flux).
+
+| File | |
+|---|---|
+| `controlplane.yaml` | Base config, **unmodified** output of `talosctl gen config` (gitignored, contains the cluster PKI) |
+| `worker.yaml` | Generated worker config, unused on a single node (gitignored) |
+| `talosconfig` | Admin client config for `talosctl` (gitignored) |
+| `patches/*.yaml` | Every change to the base config |
+
+```sh
+export TALOSCONFIG=talos/talosconfig   # run from the repo root
+N="-n 10.20.5.130 -e 10.20.5.130"
+```
+
+## Base config + patches
+
+The base file is never edited by hand; all changes live in `patches/`. The node's config is
+therefore always `controlplane.yaml` + `patches/*.yaml`, which keeps changes reviewable and lets
+a newly generated base (new node, new Talos defaults) get the same changes by reapplying the
+patches. `talosctl patch mc` merges a patch into the node's live config; it does not touch the
+local files.
+
+| Patch | Purpose |
+|---|---|
+| `control-plane-scheduling.yaml` | Drops the control-plane `NoSchedule` taint so sish can run on the only node |
+| `unprivileged-ports.yaml` | `ip_unprivileged_port_start=22`: sish (non-root, hostNetwork) binds :22/:80/:443 |
+| `firewall.yaml` | Ingress firewall, default block (see below) |
+
+Apply a single patch (dry run first; use `--mode try` for anything that can lock you out, it
+reverts automatically unless re-applied):
+
+```sh
+talosctl $N patch mc --patch @talos/patches/<patch>.yaml --mode no-reboot --dry-run
+talosctl $N patch mc --patch @talos/patches/<patch>.yaml --mode no-reboot
+```
+
+Check that the node matches the files (expect `No changes.`):
+
+```sh
+talosctl machineconfig patch talos/controlplane.yaml \
+  $(for p in talos/patches/*.yaml; do printf -- '--patch @%s ' "$p"; done) |
+  talosctl $N apply-config --file /dev/stdin --dry-run
+```
+
+## Firewall
+
+Default action `block`. Loopback and replies to outgoing connections are always allowed.
+
+| Open | Source | |
+|---|---|---|
+| 22, 80, 443, 2222, 20000-20099 tcp | anyone | sish (2222 = connector SSH, rest = forwards) |
+| 6443, 50000 tcp | anyone | Kubernetes API, Talos API (both client-cert authenticated) |
+| 53 udp/tcp | pod network `10.244.0.0/16` | CoreDNS forwards to the Talos host DNS |
+
+Closed: flannel VXLAN 4789/udp, etcd 2379-2383, kubelet 10250, kube-proxy 10256, trustd 50001
+(open it to the node network only when a second node joins).
+
+The sish ports must match `port-bind-range` in `kubernetes/base/sish/config.yml`. To add a raw
+TCP port outside 20000-20099, change both files together.
+
+## New node
+
+```sh
+talosctl gen config <cluster-name> https://<node-ip>:6443 --install-disk <disk> -o talos/
+talosctl machineconfig patch talos/controlplane.yaml \
+  $(for p in talos/patches/*.yaml; do printf -- '--patch @%s ' "$p"; done) |
+  talosctl apply-config --insecure -n <node-ip> --file /dev/stdin
+talosctl --talosconfig talos/talosconfig config endpoint <node-ip>
+talosctl --talosconfig talos/talosconfig -n <node-ip> bootstrap
+talosctl --talosconfig talos/talosconfig -n <node-ip> kubeconfig
+```
+
+Back up `controlplane.yaml` and `talosconfig` outside this folder: they are the only copy of
+the cluster PKI and admin credentials.
@@ -0,0 +1,7 @@
+# Single node: let workloads (sish) run on the control plane by dropping the
+# default control-plane NoSchedule taint.
+apiVersion: v1alpha1
+kind: KubeNodeConfig
+taints:
+  node-role.kubernetes.io/control-plane:
+    $patch: delete
@@ -0,0 +1,61 @@
+# Ingress firewall: block everything that is not listed here. Loopback and
+# replies to outgoing connections are always allowed by Talos.
+#
+# Public TCP ports served by sish must match port-bind-range in
+# kubernetes/base/sish/config.yml (plus :80 HTTP and :2222 sish SSH).
+# Closed on purpose: flannel VXLAN 4789/udp (single node, unauthenticated),
+# etcd 2379-2383, kubelet 10250, kube-proxy 10256, trustd 50001 (only needed
+# when other nodes join).
+apiVersion: v1alpha1
+kind: NetworkDefaultActionConfig
+ingress: block
+---
+apiVersion: v1alpha1
+kind: NetworkRuleConfig
+name: sish-public
+portSelector:
+  ports:
+    - 22            # gitea ssh (raw tcp forward)
+    - 80            # sish http, routed by Host header
+    - 443           # sish tls, routed by SNI
+    - 2222          # sish ssh endpoint for connectors (public key auth)
+    - 20000-20099   # reserved for raw tcp forwards
+  protocol: tcp
+ingress:
+  - subnet: 0.0.0.0/0
+  - subnet: ::/0
+---
+# Talos API (apid) and Kubernetes API, both mTLS / client-cert authenticated
+apiVersion: v1alpha1
+kind: NetworkRuleConfig
+name: talos-and-kube-api
+portSelector:
+  ports:
+    - 50000
+    - 6443
+  protocol: tcp
+ingress:
+  - subnet: 0.0.0.0/0
+  - subnet: ::/0
+---
+# CoreDNS forwards to the Talos host DNS (forwardKubeDNSToHost), which pods reach
+# on the host, so the pod network needs DNS to the node
+apiVersion: v1alpha1
+kind: NetworkRuleConfig
+name: pod-dns
+portSelector:
+  ports:
+    - 53
+  protocol: udp
+ingress:
+  - subnet: 10.244.0.0/16
+---
+apiVersion: v1alpha1
+kind: NetworkRuleConfig
+name: pod-dns-tcp
+portSelector:
+  ports:
+    - 53
+  protocol: tcp
+ingress:
+  - subnet: 10.244.0.0/16
@@ -0,0 +1,6 @@
+# Lets non-root processes bind ports >= 22, so sish (hostNetwork, uid 65534,
+# no capabilities) can listen on :22 (Gitea SSH), :80 and :443.
+# The edge runs nothing else that could grab 22-79.
+machine:
+  sysctls:
+    net.ipv4.ip_unprivileged_port_start: "22"
				`@@ -0,0 +1 @@`
				`ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOHcrMOuL1K6TPPprZi/W3N29SBRN23pAGmxp3SfQ6qr connector-dev`
				`@@ -0,0 +1 @@`
				`ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINZ1rT2t5rUS3tJesxzY7HogcBRCvzMT8HF+xNddV7e1 connector-hello`