cirrus/kubernetes

Kubernetes: sish on the edge

Plain kustomize, applied by hand. base/sish is the generic deployment, each edge gets an overlay (cirrus-dev/) with its domain, allowed hostnames, connector keys and host key.

kubectl --context admin@cirrus_dev diff -k kubernetes/cirrus-dev    # review first
kubectl --context admin@cirrus_dev apply -k kubernetes/cirrus-dev
kubectl --context admin@cirrus_dev -n sish logs deploy/sish -f

Config changes create a new ConfigMap/Secret name (kustomize hash), which rolls the pod. Rollouts use Recreate (host ports cannot be shared), so connectors drop for a few seconds and reconnect on their own.

sish

• hostNetwork, non-root (uid 65534), no capabilities, read-only root filesystem. Binding :22/:80/:443 relies on the Talos sysctl in talos/patches/unprivileged-ports.yaml.
• config.yml: SNI passthrough on :443 (sish never terminates TLS), HTTP by Host on :80, raw TCP forwards, public-key auth only, no web consoles.
• port-bind-range (ports connectors may claim) must match the firewall rule in talos/patches/firewall.yaml: 22, 443 and 20000-20099 for raw TCP forwards.
• Several connectors claiming the same host/port are load-balanced round-robin.

Overlay cirrus-dev

SISH_DOMAIN	Fallback domain sish prints for forwards
SISH_BIND_HOSTS	Parent domains connectors may claim hostnames under (exact match on everything after the first label)
pubkeys/*.pub	Authorized connector public keys, one file per connector
.secrets/ssh_host_ed25519_key	Edge SSH host key (gitignored). Connectors pin its public half (SISH_HOST_KEY)

Add a connector: put its public key into pubkeys/, list it under sish-pubkeys in kustomization.yaml, then diff and apply. Remove a connector the same way; its sessions are cut when the pod restarts.

New host key (e.g. for a new edge): ssh-keygen -t ed25519 -N '' -C sish-host@<edge> -f kubernetes/<overlay>/.secrets/ssh_host_ed25519_key, then update SISH_HOST_KEY in every connector.

Connectors run registry.traberph.de/public/sish-client (see its repo README); the cumulus cluster runs them in infra/configs/base/networking/sish-client.

sish gotchas

• port-bind-range defaults to 0,1024-65535, which rejects 22 and 443.
• A raw TCP forward must bind 0.0.0.0:<port>. With a hostname sish creates a TCP alias, reachable only through sish itself, not as a public port.
• PROXY protocol is opt-in per connector (proxy-protocol=…); the server only fixes the version.
• idle-connection-timeout defaults to 5s; raised to 1h.
• service-console-max-content-length defaults to -1, which buffers every HTTP body in memory (even with consoles off): a large upload OOM-kills sish. Set to 0 to stream.
• A name outside bind-hosts is not rejected: sish silently binds <name>.<domain> instead. Check the HTTP:/TLS: line the edge prints.
• HTTP (:80) and SNI (:443) forwards need separate connector sessions.
• Can't read file ..data log lines are harmless (Kubernetes volume symlinks).