# Kubernetes: sish on the edge

Plain kustomize, applied by hand. `base/sish` is the generic deployment, each edge gets an
overlay (`cirrus-dev/`) with its domain, allowed hostnames, connector keys and host key.

```sh
kubectl --context admin@cirrus_dev diff -k kubernetes/cirrus-dev    # review first
kubectl --context admin@cirrus_dev apply -k kubernetes/cirrus-dev
kubectl --context admin@cirrus_dev -n sish logs deploy/sish -f
```

Config changes create a new ConfigMap/Secret name (kustomize hash), which rolls the pod.
Rollouts use `Recreate` (host ports cannot be shared), so connectors drop for a few seconds
and reconnect on their own.

## sish

- `hostNetwork`, non-root (uid 65534), no capabilities, read-only root filesystem. Binding
  :22/:80/:443 relies on the Talos sysctl in `talos/patches/unprivileged-ports.yaml`.
- `config.yml`: SNI passthrough on :443 (sish never terminates TLS), HTTP by `Host` on :80,
  raw TCP forwards, public-key auth only, no web consoles.
- `port-bind-range` (ports connectors may claim) must match the firewall rule in
  `talos/patches/firewall.yaml`: 22, 443 and 20000-20099 for raw TCP forwards.
- Several connectors claiming the same host/port are load-balanced round-robin.

## Overlay `cirrus-dev`

| | |
|---|---|
| `SISH_DOMAIN` | Fallback domain sish prints for forwards |
| `SISH_BIND_HOSTS` | Parent domains connectors may claim hostnames under (exact match on everything after the first label) |
| `pubkeys/*.pub` | Authorized connector public keys, one file per connector |
| `.secrets/ssh_host_ed25519_key` | Edge SSH host key (gitignored). Connectors pin its public half (`SISH_HOST_KEY`) |

Add a connector: put its public key into `pubkeys/`, list it under `sish-pubkeys` in
`kustomization.yaml`, then diff and apply. Remove a connector the same way; its sessions are
cut when the pod restarts.

New host key (e.g. for a new edge): `ssh-keygen -t ed25519 -N '' -C sish-host@<edge> -f
kubernetes/<overlay>/.secrets/ssh_host_ed25519_key`, then update `SISH_HOST_KEY` in every
connector.

Connectors run `registry.traberph.de/public/sish-client` (see its repo README); the
`cumulus` cluster runs them in `infra/configs/base/networking/sish-client`.

## sish gotchas

- `port-bind-range` defaults to `0,1024-65535`, which rejects 22 and 443.
- A raw TCP forward must bind `0.0.0.0:<port>`. With a hostname sish creates a TCP *alias*,
  reachable only through sish itself, not as a public port.
- PROXY protocol is opt-in per connector (`proxy-protocol=…`); the server only fixes the version.
- `idle-connection-timeout` defaults to 5s; raised to 1h.
- `service-console-max-content-length` defaults to -1, which buffers every HTTP body in memory
  (even with consoles off): a large upload OOM-kills sish. Set to 0 to stream.
- A name outside `bind-hosts` is not rejected: sish silently binds `<name>.<domain>` instead.
  Check the `HTTP:`/`TLS:` line the edge prints.
- HTTP (:80) and SNI (:443) forwards need separate connector sessions.
- `Can't read file ..data` log lines are harmless (Kubernetes volume symlinks).