# Fedora CoreOS: cirrus edge (production)

Single Fedora CoreOS host that runs only sish, as a rootful Podman quadlet with host networking.
Everything is defined in `cirrus.yaml` (Butane) and applied once at install time by Ignition.
The VPS is too small for Talos, so production runs on CoreOS; the Talos setup in `talos/` and
`kubernetes/` stays the dev edge.

| File | |
|---|---|
| `cirrus.yaml` | Butane config: users, sshd, firewall, sish, updates |
| `.secrets/ssh_host_ed25519_key` | Edge SSH host key (gitignored). Connectors pin its public half (`SISH_HOST_KEY`) |
| `.secrets/connector-cumulus` | Private key of the `cumulus` connector (gitignored), goes into a Secret in `cumulus` |
| `cirrus.ign` | Build output, embeds the host key (gitignored) |

## Build and install

```sh
butane --strict --files-dir coreos coreos/cirrus.yaml > coreos/cirrus.ign
coreos-installer install /dev/<disk> --ignition-file coreos/cirrus.ign   # or the provider's user-data
```

Ignition runs only on first boot. Changing `cirrus.yaml` later does nothing to a running host:
either reinstall, or make the same change on the host by hand (and keep the file in sync).

Admin access: `ssh -p 5001 core@tunnel.traberph.de` (public key only, user `core` only).

## Network

Hostname `cirrus.traberph.de`. netcup gives IPv4 via DHCP (`46.38.234.119`) but no IPv6 router
advertisements with a usable prefix: the IPv6 address from the netcup panel (/64) is set statically
in `/etc/NetworkManager/system-connections/ens3.nmconnection`, gateway `fe80::1`.

| | |
|---|---|
| IPv4 | `46.38.234.119` (DHCP) |
| IPv6 | `2a03:4000:2:83c::1/64` (static), gateway `fe80::1` |

Only the global address (`scope global`) goes into DNS, never the `fe80::` link-local one.

## Ports

nftables (`/etc/sysconfig/nftables.conf`), default drop. Loopback, ICMP, DHCP replies and
replies to outgoing connections are allowed.

| Port (tcp) | |
|---|---|
| 5001 | Admin sshd |
| 5002 | sish SSH endpoint for connectors (public key auth) |
| 80 | sish HTTP, routed by `Host` header |
| 443 | sish TLS passthrough, routed by SNI |
| 22, 20000-20099 | Raw TCP forwards (22 = Gitea SSH) |

The forward ports must match `port-bind-range` in the sish config, otherwise a claimed port is
silently unreachable.

sshd on 5001 needs an SELinux exception (5001 is labelled `commplex_link_port_t`):
`sshd-port-selinux.service` installs `/etc/cirrus/sshd_port_5001.cil` once and again whenever an
OS update dropped it.

## sish

| Path on the host | |
|---|---|
| `/etc/containers/systemd/sish.container` | Quadlet unit (`sish.service`): non-root (uid 65532), only `CAP_NET_BIND_SERVICE`, read-only root, `MemoryMax=256M`. Own writable `/tmp` tmpfs (`Tmpfs=…,mode=1777,notmpcopyup`): sish creates a temp file per forward, and podman's automatic read-only `/tmp` (copied from the image, root 755) would make every forward fail with "remote port forwarding failed" |
| `/etc/sish/config.yml` | sish config, same as `kubernetes/base/sish/config.yml` except the values below |
| `/var/lib/sish/keys/` | Host key (read-only in the container) |
| `/var/lib/sish/pubkeys/clients` | Authorized connector keys, `authorized_keys` format |

Differences to the k8s config:

| | |
|---|---|
| `ssh-address: ":5002"` | 2222 in k8s |
| `domain: tunnel.traberph.de` | The edge's own name. A requested name without a dot becomes `<name>.tunnel.traberph.de` |
| `bind-any-host: true` | Single tenant: connectors may claim any hostname containing a dot, wildcards included. Replaces `bind-hosts` |
| `verify-dns: false` | No `_sish` TXT ownership checks (pointless with `bind-any-host`) |

Every authorized key can claim every hostname and port, and with the load balancers on it can join
an existing one. So only add keys of connectors you control (sish has no per-key permissions).

**Add or remove a connector:** edit `/var/lib/sish/pubkeys/clients` on the host (sish watches the
directory, no restart needed) and the same block in `cirrus.yaml`.

**Config change:** edit `/etc/sish/config.yml`, `systemctl restart sish`, mirror it in
`cirrus.yaml`. Connectors drop for a few seconds and reconnect on their own.

```sh
systemctl status sish
journalctl -u sish -f
```

## DNS

| Record | |
|---|---|
| `tunnel.traberph.de` A/AAAA → VPS | Connectors (:5002) and admin SSH (:5001) |
| `<host>` or `*.<domain>` A/AAAA → VPS | Every hostname a connector serves; unclaimed names get a 404 (:80) or no answer (:443) |
| `*.tunnel.traberph.de` A/AAAA → VPS | Optional, only if fallback names should be reachable |

A wildcard claim (`*.example.com`) does not cover the apex `example.com`, neither in DNS nor in sish:
connectors claim the apex separately. Records that point elsewhere (e.g. still proxied through
Cloudflare) take precedence over the wildcard; deleting such a record silently moves the name to the
edge, where it only works if a connector serves it.

## Updates

Both are automatic:

- **OS:** Zincati stages new Fedora CoreOS releases (stable stream) and reboots only in the window
  03:00-04:00 UTC (`/etc/zincati/config.d/55-updates-strategy.toml`).
- **sish:** upstream publishes only exact tags (`v2.24.0`), so `podman auto-update` can't follow a
  version line. `sish-update.timer` (daily ~05:00 UTC) runs `/usr/local/bin/sish-update`, which
  sets `Image=` in the quadlet to the newest tag matching `TRACK=v2.` (minor + patch, never a new
  major), restarts sish and rolls back if nothing listens on :5002 after 30s. `TRACK=v2.24.` limits
  it to patch releases.

```sh
journalctl -u zincati -u sish-update
systemctl start sish-update   # check now
```

Both restart sish (tunnels drop for a few seconds). A major sish release (`v3`) is a manual change
of `TRACK` and `Image=`.

## First boot checklist

- `ss -tlnp`: sish on 22, 80, 443, 5002; sshd on 5001.
- `journalctl -u sish`: `Loading ssh_host_ed25519_key as ssh-ed25519 host key` (the provisioned
  key, not a generated one).
- After the first OS update: SSH on 5001 still works, `semodule -l | grep sshd_port_5001`.