ipv6
This commit is contained in:
@@ -0,0 +1,130 @@
|
||||
# Fedora CoreOS: cirrus edge (production)
|
||||
|
||||
Single Fedora CoreOS host that runs only sish, as a rootful Podman quadlet with host networking.
|
||||
Everything is defined in `cirrus.yaml` (Butane) and applied once at install time by Ignition.
|
||||
The VPS is too small for Talos, so production runs on CoreOS; the Talos setup in `talos/` and
|
||||
`kubernetes/` stays the dev edge.
|
||||
|
||||
| File | |
|
||||
|---|---|
|
||||
| `cirrus.yaml` | Butane config: users, sshd, firewall, sish, updates |
|
||||
| `.secrets/ssh_host_ed25519_key` | Edge SSH host key (gitignored). Connectors pin its public half (`SISH_HOST_KEY`) |
|
||||
| `.secrets/connector-cumulus` | Private key of the `cumulus` connector (gitignored), goes into a Secret in `cumulus` |
|
||||
| `cirrus.ign` | Build output, embeds the host key (gitignored) |
|
||||
|
||||
## Build and install
|
||||
|
||||
```sh
|
||||
butane --strict --files-dir coreos coreos/cirrus.yaml > coreos/cirrus.ign
|
||||
coreos-installer install /dev/<disk> --ignition-file coreos/cirrus.ign # or the provider's user-data
|
||||
```
|
||||
|
||||
Ignition runs only on first boot. Changing `cirrus.yaml` later does nothing to a running host:
|
||||
either reinstall, or make the same change on the host by hand (and keep the file in sync).
|
||||
|
||||
Admin access: `ssh -p 5001 core@tunnel.traberph.de` (public key only, user `core` only).
|
||||
|
||||
## Network
|
||||
|
||||
Hostname `cirrus.traberph.de`. netcup gives IPv4 via DHCP (`46.38.234.119`) but no IPv6 router
|
||||
advertisements with a usable prefix: the IPv6 address from the netcup panel (/64) is set statically
|
||||
in `/etc/NetworkManager/system-connections/ens3.nmconnection`, gateway `fe80::1`.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| IPv4 | `46.38.234.119` (DHCP) |
|
||||
| IPv6 | `2a03:4000:2:83c::1/64` (static), gateway `fe80::1` |
|
||||
|
||||
Only the global address (`scope global`) goes into DNS, never the `fe80::` link-local one.
|
||||
|
||||
## Ports
|
||||
|
||||
nftables (`/etc/sysconfig/nftables.conf`), default drop. Loopback, ICMP, DHCP replies and
|
||||
replies to outgoing connections are allowed.
|
||||
|
||||
| Port (tcp) | |
|
||||
|---|---|
|
||||
| 5001 | Admin sshd |
|
||||
| 5002 | sish SSH endpoint for connectors (public key auth) |
|
||||
| 80 | sish HTTP, routed by `Host` header |
|
||||
| 443 | sish TLS passthrough, routed by SNI |
|
||||
| 22, 20000-20099 | Raw TCP forwards (22 = Gitea SSH) |
|
||||
|
||||
The forward ports must match `port-bind-range` in the sish config, otherwise a claimed port is
|
||||
silently unreachable.
|
||||
|
||||
sshd on 5001 needs an SELinux exception (5001 is labelled `commplex_link_port_t`):
|
||||
`sshd-port-selinux.service` installs `/etc/cirrus/sshd_port_5001.cil` once and again whenever an
|
||||
OS update dropped it.
|
||||
|
||||
## sish
|
||||
|
||||
| Path on the host | |
|
||||
|---|---|
|
||||
| `/etc/containers/systemd/sish.container` | Quadlet unit (`sish.service`): non-root (uid 65532), only `CAP_NET_BIND_SERVICE`, read-only root, `MemoryMax=256M`. Own writable `/tmp` tmpfs (`Tmpfs=…,mode=1777,notmpcopyup`): sish creates a temp file per forward, and podman's automatic read-only `/tmp` (copied from the image, root 755) would make every forward fail with "remote port forwarding failed" |
|
||||
| `/etc/sish/config.yml` | sish config, same as `kubernetes/base/sish/config.yml` except the values below |
|
||||
| `/var/lib/sish/keys/` | Host key (read-only in the container) |
|
||||
| `/var/lib/sish/pubkeys/clients` | Authorized connector keys, `authorized_keys` format |
|
||||
|
||||
Differences to the k8s config:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `ssh-address: ":5002"` | 2222 in k8s |
|
||||
| `domain: tunnel.traberph.de` | The edge's own name. A requested name without a dot becomes `<name>.tunnel.traberph.de` |
|
||||
| `bind-any-host: true` | Single tenant: connectors may claim any hostname containing a dot, wildcards included. Replaces `bind-hosts` |
|
||||
| `verify-dns: false` | No `_sish` TXT ownership checks (pointless with `bind-any-host`) |
|
||||
|
||||
Every authorized key can claim every hostname and port, and with the load balancers on it can join
|
||||
an existing one. So only add keys of connectors you control (sish has no per-key permissions).
|
||||
|
||||
**Add or remove a connector:** edit `/var/lib/sish/pubkeys/clients` on the host (sish watches the
|
||||
directory, no restart needed) and the same block in `cirrus.yaml`.
|
||||
|
||||
**Config change:** edit `/etc/sish/config.yml`, `systemctl restart sish`, mirror it in
|
||||
`cirrus.yaml`. Connectors drop for a few seconds and reconnect on their own.
|
||||
|
||||
```sh
|
||||
systemctl status sish
|
||||
journalctl -u sish -f
|
||||
```
|
||||
|
||||
## DNS
|
||||
|
||||
| Record | |
|
||||
|---|---|
|
||||
| `tunnel.traberph.de` A/AAAA → VPS | Connectors (:5002) and admin SSH (:5001) |
|
||||
| `<host>` or `*.<domain>` A/AAAA → VPS | Every hostname a connector serves; unclaimed names get a 404 (:80) or no answer (:443) |
|
||||
| `*.tunnel.traberph.de` A/AAAA → VPS | Optional, only if fallback names should be reachable |
|
||||
|
||||
A wildcard claim (`*.example.com`) does not cover the apex `example.com`, neither in DNS nor in sish:
|
||||
connectors claim the apex separately. Records that point elsewhere (e.g. still proxied through
|
||||
Cloudflare) take precedence over the wildcard; deleting such a record silently moves the name to the
|
||||
edge, where it only works if a connector serves it.
|
||||
|
||||
## Updates
|
||||
|
||||
Both are automatic:
|
||||
|
||||
- **OS:** Zincati stages new Fedora CoreOS releases (stable stream) and reboots only in the window
|
||||
03:00-04:00 UTC (`/etc/zincati/config.d/55-updates-strategy.toml`).
|
||||
- **sish:** upstream publishes only exact tags (`v2.24.0`), so `podman auto-update` can't follow a
|
||||
version line. `sish-update.timer` (daily ~05:00 UTC) runs `/usr/local/bin/sish-update`, which
|
||||
sets `Image=` in the quadlet to the newest tag matching `TRACK=v2.` (minor + patch, never a new
|
||||
major), restarts sish and rolls back if nothing listens on :5002 after 30s. `TRACK=v2.24.` limits
|
||||
it to patch releases.
|
||||
|
||||
```sh
|
||||
journalctl -u zincati -u sish-update
|
||||
systemctl start sish-update # check now
|
||||
```
|
||||
|
||||
Both restart sish (tunnels drop for a few seconds). A major sish release (`v3`) is a manual change
|
||||
of `TRACK` and `Image=`.
|
||||
|
||||
## First boot checklist
|
||||
|
||||
- `ss -tlnp`: sish on 22, 80, 443, 5002; sshd on 5001.
|
||||
- `journalctl -u sish`: `Loading ssh_host_ed25519_key as ssh-ed25519 host key` (the provisioned
|
||||
key, not a generated one).
|
||||
- After the first OS update: SSH on 5001 still works, `semodule -l | grep sshd_port_5001`.
|
||||
@@ -16,6 +16,27 @@ storage:
|
||||
user: { id: 65532 }
|
||||
group: { id: 65532 }
|
||||
files:
|
||||
- path: /etc/hostname
|
||||
mode: 0644
|
||||
contents:
|
||||
inline: cirrus.traberph.de
|
||||
# netcup: IPv4 via DHCP, IPv6 static (no router advertisements, gateway is always fe80::1)
|
||||
- path: /etc/NetworkManager/system-connections/ens3.nmconnection
|
||||
mode: 0600
|
||||
contents:
|
||||
inline: |
|
||||
[connection]
|
||||
id=ens3
|
||||
type=ethernet
|
||||
interface-name=ens3
|
||||
|
||||
[ipv4]
|
||||
method=auto
|
||||
|
||||
[ipv6]
|
||||
method=manual
|
||||
address1=2a03:4000:2:83c::1/64
|
||||
gateway=fe80::1
|
||||
# Edge SSH host key (connectors pin its public half). Kept only locally in coreos/.secrets/
|
||||
# (gitignored), so back it up outside this folder. Build: butane --files-dir coreos ...
|
||||
- path: /var/lib/sish/keys/ssh_host_ed25519_key
|
||||
@@ -118,6 +139,9 @@ storage:
|
||||
AddCapability=CAP_NET_BIND_SERVICE
|
||||
NoNewPrivileges=true
|
||||
ReadOnly=true
|
||||
# sish creates a temp file per forward. The automatic read-only /tmp tmpfs copies the
|
||||
# image's /tmp (root, 755), so uid 65532 can't write there: give it a plain sticky /tmp.
|
||||
Tmpfs=/tmp:rw,nosuid,nodev,noexec,size=16m,mode=1777,notmpcopyup
|
||||
Volume=/etc/sish:/config:ro,Z
|
||||
Volume=/var/lib/sish/keys:/keys:ro,Z
|
||||
Volume=/var/lib/sish/pubkeys:/pubkeys:ro,Z
|
||||
|
||||
Reference in New Issue
Block a user