59 lines
3.0 KiB
Markdown
59 lines
3.0 KiB
Markdown
# Kubernetes: sish on the edge
|
|
|
|
Plain kustomize, applied by hand. `base/sish` is the generic deployment, each edge gets an
|
|
overlay (`cirrus-dev/`) with its domain, allowed hostnames, connector keys and host key.
|
|
|
|
```sh
|
|
kubectl --context admin@cirrus_dev diff -k kubernetes/cirrus-dev # review first
|
|
kubectl --context admin@cirrus_dev apply -k kubernetes/cirrus-dev
|
|
kubectl --context admin@cirrus_dev -n sish logs deploy/sish -f
|
|
```
|
|
|
|
Config changes create a new ConfigMap/Secret name (kustomize hash), which rolls the pod.
|
|
Rollouts use `Recreate` (host ports cannot be shared), so connectors drop for a few seconds
|
|
and reconnect on their own.
|
|
|
|
## sish
|
|
|
|
- `hostNetwork`, non-root (uid 65534), no capabilities, read-only root filesystem. Binding
|
|
:22/:80/:443 relies on the Talos sysctl in `talos/patches/unprivileged-ports.yaml`.
|
|
- `config.yml`: SNI passthrough on :443 (sish never terminates TLS), HTTP by `Host` on :80,
|
|
raw TCP forwards, public-key auth only, no web consoles.
|
|
- `port-bind-range` (ports connectors may claim) must match the firewall rule in
|
|
`talos/patches/firewall.yaml`: 22, 443 and 20000-20099 for raw TCP forwards.
|
|
- Several connectors claiming the same host/port are load-balanced round-robin.
|
|
|
|
## Overlay `cirrus-dev`
|
|
|
|
| | |
|
|
|---|---|
|
|
| `SISH_DOMAIN` | Fallback domain sish prints for forwards |
|
|
| `SISH_BIND_HOSTS` | Parent domains connectors may claim hostnames under (exact match on everything after the first label) |
|
|
| `pubkeys/*.pub` | Authorized connector public keys, one file per connector |
|
|
| `.secrets/ssh_host_ed25519_key` | Edge SSH host key (gitignored). Connectors pin its public half (`SISH_HOST_KEY`) |
|
|
|
|
Add a connector: put its public key into `pubkeys/`, list it under `sish-pubkeys` in
|
|
`kustomization.yaml`, then diff and apply. Remove a connector the same way; its sessions are
|
|
cut when the pod restarts.
|
|
|
|
New host key (e.g. for a new edge): `ssh-keygen -t ed25519 -N '' -C sish-host@<edge> -f
|
|
kubernetes/<overlay>/.secrets/ssh_host_ed25519_key`, then update `SISH_HOST_KEY` in every
|
|
connector.
|
|
|
|
Connectors run `registry.traberph.de/public/sish-client` (see its repo README); the
|
|
`cumulus` cluster runs them in `infra/configs/base/networking/sish-client`.
|
|
|
|
## sish gotchas
|
|
|
|
- `port-bind-range` defaults to `0,1024-65535`, which rejects 22 and 443.
|
|
- A raw TCP forward must bind `0.0.0.0:<port>`. With a hostname sish creates a TCP *alias*,
|
|
reachable only through sish itself, not as a public port.
|
|
- PROXY protocol is opt-in per connector (`proxy-protocol=…`); the server only fixes the version.
|
|
- `idle-connection-timeout` defaults to 5s; raised to 1h.
|
|
- `service-console-max-content-length` defaults to -1, which buffers every HTTP body in memory
|
|
(even with consoles off): a large upload OOM-kills sish. Set to 0 to stream.
|
|
- A name outside `bind-hosts` is not rejected: sish silently binds `<name>.<domain>` instead.
|
|
Check the `HTTP:`/`TLS:` line the edge prints.
|
|
- HTTP (:80) and SNI (:443) forwards need separate connector sessions.
|
|
- `Can't read file ..data` log lines are harmless (Kubernetes volume symlinks).
|