cleanup talos
This commit is contained in:
+5
-13
@@ -1,16 +1,8 @@
|
||||
# Plaintext secrets: never commit, back them up outside this folder
|
||||
# Plaintext secrets (edge host key, connector keys): never commit, back them up outside this folder
|
||||
.secrets/
|
||||
|
||||
# Talos generated configs contain cluster PKI and admin credentials (any folder,
|
||||
# e.g. a new edge's config generated next to talos/)
|
||||
controlplane.yaml
|
||||
worker.yaml
|
||||
talosconfig
|
||||
secrets.yaml
|
||||
kubeconfig
|
||||
|
||||
# Retired dev edge credentials (cirrus_dev), kept locally only
|
||||
old/
|
||||
|
||||
# Ignition output embeds the secrets above
|
||||
# Ignition output embeds the host key from .secrets/
|
||||
*.ign
|
||||
|
||||
# Retired setups (Talos dev edge, Kubernetes) and their credentials, kept locally only
|
||||
old/
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# cirrus
|
||||
|
||||
Self-hosted edge: a host running only [sish](https://github.com/antoniomika/sish). Clusters
|
||||
without inbound ports (e.g. `cumulus`) open outbound SSH tunnels to it with `sish-client`, and the
|
||||
edge relays public traffic back through them:
|
||||
Self-hosted edge: a Fedora CoreOS VPS running only [sish](https://github.com/antoniomika/sish).
|
||||
Clusters without inbound ports (e.g. `cumulus`) open outbound SSH tunnels to it with `sish-client`,
|
||||
and the edge relays public traffic back through them:
|
||||
|
||||
```
|
||||
client ──▶ edge :22/:80/:443/:200xx (sish) ══ssh══▶ sish-client ──▶ envoy gateway ──▶ app
|
||||
@@ -11,46 +11,145 @@ client ──▶ edge :22/:80/:443/:200xx (sish) ══ssh══▶ sish-client
|
||||
- :443 is routed by SNI without decrypting (TLS passthrough), optionally with a PROXY v2 header.
|
||||
- :80 is routed by `Host` header, raw TCP ports (e.g. :22 for Gitea SSH) by port.
|
||||
|
||||
Production runs on Fedora CoreOS (the VPS is too small for Talos), the dev edge on Talos +
|
||||
Kubernetes. Both use the same sish configuration.
|
||||
Production serves everything from `cumulus`: `traberph.de` and `*.traberph.de` on :80/:443 (IPv4
|
||||
and IPv6), Gitea SSH on :22. The cluster side (connectors, Envoy gateways, per-app routes) is
|
||||
documented in the `cumulus` README.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
coreos/ production edge: Butane config (sish quadlet, firewall, updates) → coreos/README.md
|
||||
talos/ dev edge node config: generated base + patches → talos/README.md
|
||||
kubernetes/ dev edge sish deployment, kustomize base + overlay → kubernetes/README.md
|
||||
**/.secrets/ host and connector private keys (gitignored)
|
||||
| File | |
|
||||
|---|---|
|
||||
| `cirrus.yaml` | Butane config: users, sshd, network, firewall, sish, updates |
|
||||
| `.secrets/ssh_host_ed25519_key` | Edge SSH host key (gitignored). Connectors pin its public half (`SISH_HOST_KEY`) |
|
||||
| `.secrets/connector-cumulus` | Private key of the `cumulus` connector (gitignored), goes into a Secret in `cumulus` |
|
||||
| `cirrus.ign` | Build output, embeds the host key (gitignored) |
|
||||
|
||||
Secrets only live in the gitignored `.secrets/` and `*.ign`; nothing secret is committed. Back up
|
||||
`.secrets/` outside this folder (password manager), it is the only copy.
|
||||
|
||||
## Build and install
|
||||
|
||||
```sh
|
||||
butane --strict --files-dir . cirrus.yaml > cirrus.ign
|
||||
coreos-installer install /dev/<disk> --ignition-file cirrus.ign # or the provider's user-data
|
||||
```
|
||||
|
||||
Everything is applied by hand (`butane` + Ignition, `talosctl`, `kubectl apply -k`). Secrets never
|
||||
leave the gitignored files (`coreos/.secrets/`, `coreos/*.ign`, `talos/controlplane.yaml`,
|
||||
`talos/talosconfig`, `**/.secrets/`).
|
||||
Ignition runs only on first boot. Changing `cirrus.yaml` later does nothing to a running host:
|
||||
either reinstall, or make the same change on the host by hand (and keep the file in sync).
|
||||
|
||||
## Environments
|
||||
Admin access: `ssh -p 5001 core@tunnel.traberph.de` (public key only, user `core` only).
|
||||
|
||||
| | Edge | Domains |
|
||||
|---|---|---|
|
||||
| `cirrus` | `tunnel.traberph.de`, Fedora CoreOS (stable) | any hostname pointed at the VPS |
|
||||
| `cirrus-dev` | `10.20.5.130` (LAN), Talos v1.14.1, Kubernetes v1.37.0 | `.test` / `.sto` via local DNS |
|
||||
### First boot checklist
|
||||
|
||||
Production (`cirrus`) serves everything from `cumulus` (since 2026-10-06): `traberph.de` and
|
||||
`*.traberph.de` on :80/:443 (IPv4 and IPv6), Gitea SSH on :22. The cluster side (connectors, Envoy
|
||||
gateways, per-app routes) is documented in the `cumulus` README. The dev edge only serves
|
||||
`.test`/`.sto` names.
|
||||
- `ss -tlnp`: sish on 22, 80, 443, 5002; sshd on 5001.
|
||||
- `journalctl -u sish`: `Loading ssh_host_ed25519_key as ssh-ed25519 host key` (the provisioned
|
||||
key, not a generated one).
|
||||
- After the first OS update: SSH on 5001 still works, `semodule -l | grep sshd_port_5001`.
|
||||
|
||||
## Production rollout (done 2026-10-06)
|
||||
## Network
|
||||
|
||||
Rolled out as planned: CoreOS VPS with sish, second connector on `cumulus` for TLS passthrough +
|
||||
PROXY v2, Envoy HTTPS gateway with cert-manager (Let's Encrypt HTTP-01 over the :80 route), services
|
||||
moved from the Cloudflare tunnel one hostname at a time by switching DNS.
|
||||
Hostname `cirrus.traberph.de` on netcup. netcup gives IPv4 via DHCP but no IPv6 router
|
||||
advertisements with a usable prefix: the IPv6 address from the netcup panel (/64) is set statically
|
||||
in `/etc/NetworkManager/system-connections/ens3.nmconnection`, gateway `fe80::1`.
|
||||
|
||||
**Still open**
|
||||
- **Backups.** `coreos/.secrets/` (edge host key, `cumulus` connector key) and the Talos dev
|
||||
credentials exist only in this folder. Store them in a password manager.
|
||||
| | |
|
||||
|---|---|
|
||||
| IPv4 | `46.38.234.119` (DHCP) |
|
||||
| IPv6 | `2a03:4000:2:83c::1/64` (static), gateway `fe80::1` |
|
||||
|
||||
Only the global address (`scope global`) goes into DNS, never the `fe80::` link-local one.
|
||||
|
||||
## Ports
|
||||
|
||||
nftables (`/etc/sysconfig/nftables.conf`), default drop. Loopback, ICMP, DHCP replies and
|
||||
replies to outgoing connections are allowed.
|
||||
|
||||
| Port (tcp) | |
|
||||
|---|---|
|
||||
| 5001 | Admin sshd |
|
||||
| 5002 | sish SSH endpoint for connectors (public key auth) |
|
||||
| 80 | sish HTTP, routed by `Host` header |
|
||||
| 443 | sish TLS passthrough, routed by SNI |
|
||||
| 22, 20000-20099 | Raw TCP forwards (22 = Gitea SSH) |
|
||||
|
||||
The forward ports must match `port-bind-range` in the sish config, otherwise a claimed port is
|
||||
silently unreachable.
|
||||
|
||||
sshd on 5001 needs an SELinux exception (5001 is labelled `commplex_link_port_t`):
|
||||
`sshd-port-selinux.service` installs `/etc/cirrus/sshd_port_5001.cil` once and again whenever an
|
||||
OS update dropped it.
|
||||
|
||||
## sish
|
||||
|
||||
| Path on the host | |
|
||||
|---|---|
|
||||
| `/etc/containers/systemd/sish.container` | Quadlet unit (`sish.service`), rootful Podman with host networking: non-root (uid 65532), only `CAP_NET_BIND_SERVICE`, read-only root, `MemoryMax=256M`. Own writable `/tmp` tmpfs (`Tmpfs=…,mode=1777,notmpcopyup`): sish creates a temp file per forward, and podman's automatic read-only `/tmp` (copied from the image, root 755) would make every forward fail with "remote port forwarding failed" |
|
||||
| `/etc/sish/config.yml` | sish config |
|
||||
| `/var/lib/sish/keys/` | Host key (read-only in the container) |
|
||||
| `/var/lib/sish/pubkeys/clients` | Authorized connector keys, `authorized_keys` format |
|
||||
|
||||
Notable config values:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `ssh-address: ":5002"` | Connector SSH endpoint |
|
||||
| `domain: tunnel.traberph.de` | The edge's own name. A requested name without a dot becomes `<name>.tunnel.traberph.de` |
|
||||
| `bind-any-host: true` | Single tenant: connectors may claim any hostname containing a dot, wildcards included |
|
||||
| `verify-dns: false` | No `_sish` TXT ownership checks (pointless with `bind-any-host`) |
|
||||
| `sni-proxy`, `*-load-balancer: true` | TLS passthrough on :443, several connectors may serve the same name |
|
||||
| `proxy-protocol-version: "2"` | PROXY header for connectors that request it |
|
||||
| `idle-connection-timeout: 1h` | Default 5s kills websockets, SSE and slow uploads |
|
||||
| `service-console-max-content-length: 0` | Default -1 buffers every body in memory, large uploads OOM-kill sish |
|
||||
|
||||
Every authorized key can claim every hostname and port, and with the load balancers on it can join
|
||||
an existing one. So only add keys of connectors you control (sish has no per-key permissions).
|
||||
|
||||
**Add or remove a connector:** edit `/var/lib/sish/pubkeys/clients` on the host (sish watches the
|
||||
directory, no restart needed) and the same block in `cirrus.yaml`.
|
||||
|
||||
**Config change:** edit `/etc/sish/config.yml`, `systemctl restart sish`, mirror it in
|
||||
`cirrus.yaml`. Connectors drop for a few seconds and reconnect on their own.
|
||||
|
||||
```sh
|
||||
systemctl status sish
|
||||
journalctl -u sish -f
|
||||
```
|
||||
|
||||
## DNS
|
||||
|
||||
| Record | |
|
||||
|---|---|
|
||||
| `tunnel.traberph.de` A/AAAA → VPS | Connectors (:5002) and admin SSH (:5001) |
|
||||
| `<host>` or `*.<domain>` A/AAAA → VPS | Every hostname a connector serves; unclaimed names get a 404 (:80) or no answer (:443) |
|
||||
| `*.tunnel.traberph.de` A/AAAA → VPS | Optional, only if fallback names should be reachable |
|
||||
|
||||
A wildcard claim (`*.example.com`) does not cover the apex `example.com`, neither in DNS nor in sish:
|
||||
connectors claim the apex separately. Records that point elsewhere (e.g. still proxied through
|
||||
Cloudflare) take precedence over the wildcard; deleting such a record silently moves the name to the
|
||||
edge, where it only works if a connector serves it.
|
||||
|
||||
## Updates
|
||||
|
||||
Both are automatic:
|
||||
|
||||
- **OS:** Zincati stages new Fedora CoreOS releases (stable stream) and reboots only in the window
|
||||
03:00-04:00 UTC (`/etc/zincati/config.d/55-updates-strategy.toml`).
|
||||
- **sish:** upstream publishes only exact tags (`v2.24.0`), so `podman auto-update` can't follow a
|
||||
version line. `sish-update.timer` (daily ~05:00 UTC) runs `/usr/local/bin/sish-update`, which
|
||||
sets `Image=` in the quadlet to the newest tag matching `TRACK=v2.` (minor + patch, never a new
|
||||
major), restarts sish and rolls back if nothing listens on :5002 after 30s. `TRACK=v2.24.` limits
|
||||
it to patch releases.
|
||||
|
||||
```sh
|
||||
journalctl -u zincati -u sish-update
|
||||
systemctl start sish-update # check now
|
||||
```
|
||||
|
||||
Both restart sish (tunnels drop for a few seconds). A major sish release (`v3`) is a manual change
|
||||
of `TRACK` and `Image=`. The `sish-client` tags in `cumulus` are updated by hand.
|
||||
|
||||
## Open
|
||||
|
||||
- **Backups.** `.secrets/` exists only in this folder. Store it in a password manager.
|
||||
- **Uptime check.** External check on the edge (sish :5002 and one route per protocol): the edge is
|
||||
a single point of failure for everything behind it.
|
||||
- **Updates by hand.** Production updates are automatic (OS in a nightly reboot window, sish within
|
||||
v2, see `coreos/README.md`). sish-client tags in `cumulus` and the dev edge (`talosctl upgrade` /
|
||||
`upgrade-k8s`, sish image tag) are updated by hand; the `talosconfig` admin certificate expires
|
||||
after one year.
|
||||
|
||||
Executable → Regular
+3
-5
@@ -37,15 +37,15 @@ storage:
|
||||
method=manual
|
||||
address1=2a03:4000:2:83c::1/64
|
||||
gateway=fe80::1
|
||||
# Edge SSH host key (connectors pin its public half). Kept only locally in coreos/.secrets/
|
||||
# (gitignored), so back it up outside this folder. Build: butane --files-dir coreos ...
|
||||
# Edge SSH host key (connectors pin its public half). Kept only locally in .secrets/
|
||||
# (gitignored), so back it up outside this folder. Build: butane --files-dir . ...
|
||||
- path: /var/lib/sish/keys/ssh_host_ed25519_key
|
||||
mode: 0400
|
||||
user: { id: 65532 }
|
||||
group: { id: 65532 }
|
||||
contents:
|
||||
local: .secrets/ssh_host_ed25519_key
|
||||
- path: /var/lib/sish/pubkeys/clients # k8s tunnel client keys (authorized_keys format)
|
||||
- path: /var/lib/sish/pubkeys/clients # connector public keys (authorized_keys format)
|
||||
mode: 0644
|
||||
contents:
|
||||
inline: |
|
||||
@@ -63,8 +63,6 @@ storage:
|
||||
contents:
|
||||
inline: |
|
||||
(allow sshd_t commplex_link_port_t (tcp_socket (name_bind)))
|
||||
# Same sish config as kubernetes/base/sish/config.yml (+ the cirrus overlay values),
|
||||
# only the SSH port differs (5002 instead of 2222).
|
||||
- path: /etc/sish/config.yml
|
||||
mode: 0644
|
||||
contents:
|
||||
@@ -1,130 +0,0 @@
|
||||
# Fedora CoreOS: cirrus edge (production)
|
||||
|
||||
Single Fedora CoreOS host that runs only sish, as a rootful Podman quadlet with host networking.
|
||||
Everything is defined in `cirrus.yaml` (Butane) and applied once at install time by Ignition.
|
||||
The VPS is too small for Talos, so production runs on CoreOS; the Talos setup in `talos/` and
|
||||
`kubernetes/` stays the dev edge.
|
||||
|
||||
| File | |
|
||||
|---|---|
|
||||
| `cirrus.yaml` | Butane config: users, sshd, firewall, sish, updates |
|
||||
| `.secrets/ssh_host_ed25519_key` | Edge SSH host key (gitignored). Connectors pin its public half (`SISH_HOST_KEY`) |
|
||||
| `.secrets/connector-cumulus` | Private key of the `cumulus` connector (gitignored), goes into a Secret in `cumulus` |
|
||||
| `cirrus.ign` | Build output, embeds the host key (gitignored) |
|
||||
|
||||
## Build and install
|
||||
|
||||
```sh
|
||||
butane --strict --files-dir coreos coreos/cirrus.yaml > coreos/cirrus.ign
|
||||
coreos-installer install /dev/<disk> --ignition-file coreos/cirrus.ign # or the provider's user-data
|
||||
```
|
||||
|
||||
Ignition runs only on first boot. Changing `cirrus.yaml` later does nothing to a running host:
|
||||
either reinstall, or make the same change on the host by hand (and keep the file in sync).
|
||||
|
||||
Admin access: `ssh -p 5001 core@tunnel.traberph.de` (public key only, user `core` only).
|
||||
|
||||
## Network
|
||||
|
||||
Hostname `cirrus.traberph.de`. netcup gives IPv4 via DHCP (`46.38.234.119`) but no IPv6 router
|
||||
advertisements with a usable prefix: the IPv6 address from the netcup panel (/64) is set statically
|
||||
in `/etc/NetworkManager/system-connections/ens3.nmconnection`, gateway `fe80::1`.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| IPv4 | `46.38.234.119` (DHCP) |
|
||||
| IPv6 | `2a03:4000:2:83c::1/64` (static), gateway `fe80::1` |
|
||||
|
||||
Only the global address (`scope global`) goes into DNS, never the `fe80::` link-local one.
|
||||
|
||||
## Ports
|
||||
|
||||
nftables (`/etc/sysconfig/nftables.conf`), default drop. Loopback, ICMP, DHCP replies and
|
||||
replies to outgoing connections are allowed.
|
||||
|
||||
| Port (tcp) | |
|
||||
|---|---|
|
||||
| 5001 | Admin sshd |
|
||||
| 5002 | sish SSH endpoint for connectors (public key auth) |
|
||||
| 80 | sish HTTP, routed by `Host` header |
|
||||
| 443 | sish TLS passthrough, routed by SNI |
|
||||
| 22, 20000-20099 | Raw TCP forwards (22 = Gitea SSH) |
|
||||
|
||||
The forward ports must match `port-bind-range` in the sish config, otherwise a claimed port is
|
||||
silently unreachable.
|
||||
|
||||
sshd on 5001 needs an SELinux exception (5001 is labelled `commplex_link_port_t`):
|
||||
`sshd-port-selinux.service` installs `/etc/cirrus/sshd_port_5001.cil` once and again whenever an
|
||||
OS update dropped it.
|
||||
|
||||
## sish
|
||||
|
||||
| Path on the host | |
|
||||
|---|---|
|
||||
| `/etc/containers/systemd/sish.container` | Quadlet unit (`sish.service`): non-root (uid 65532), only `CAP_NET_BIND_SERVICE`, read-only root, `MemoryMax=256M`. Own writable `/tmp` tmpfs (`Tmpfs=…,mode=1777,notmpcopyup`): sish creates a temp file per forward, and podman's automatic read-only `/tmp` (copied from the image, root 755) would make every forward fail with "remote port forwarding failed" |
|
||||
| `/etc/sish/config.yml` | sish config, same as `kubernetes/base/sish/config.yml` except the values below |
|
||||
| `/var/lib/sish/keys/` | Host key (read-only in the container) |
|
||||
| `/var/lib/sish/pubkeys/clients` | Authorized connector keys, `authorized_keys` format |
|
||||
|
||||
Differences to the k8s config:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `ssh-address: ":5002"` | 2222 in k8s |
|
||||
| `domain: tunnel.traberph.de` | The edge's own name. A requested name without a dot becomes `<name>.tunnel.traberph.de` |
|
||||
| `bind-any-host: true` | Single tenant: connectors may claim any hostname containing a dot, wildcards included. Replaces `bind-hosts` |
|
||||
| `verify-dns: false` | No `_sish` TXT ownership checks (pointless with `bind-any-host`) |
|
||||
|
||||
Every authorized key can claim every hostname and port, and with the load balancers on it can join
|
||||
an existing one. So only add keys of connectors you control (sish has no per-key permissions).
|
||||
|
||||
**Add or remove a connector:** edit `/var/lib/sish/pubkeys/clients` on the host (sish watches the
|
||||
directory, no restart needed) and the same block in `cirrus.yaml`.
|
||||
|
||||
**Config change:** edit `/etc/sish/config.yml`, `systemctl restart sish`, mirror it in
|
||||
`cirrus.yaml`. Connectors drop for a few seconds and reconnect on their own.
|
||||
|
||||
```sh
|
||||
systemctl status sish
|
||||
journalctl -u sish -f
|
||||
```
|
||||
|
||||
## DNS
|
||||
|
||||
| Record | |
|
||||
|---|---|
|
||||
| `tunnel.traberph.de` A/AAAA → VPS | Connectors (:5002) and admin SSH (:5001) |
|
||||
| `<host>` or `*.<domain>` A/AAAA → VPS | Every hostname a connector serves; unclaimed names get a 404 (:80) or no answer (:443) |
|
||||
| `*.tunnel.traberph.de` A/AAAA → VPS | Optional, only if fallback names should be reachable |
|
||||
|
||||
A wildcard claim (`*.example.com`) does not cover the apex `example.com`, neither in DNS nor in sish:
|
||||
connectors claim the apex separately. Records that point elsewhere (e.g. still proxied through
|
||||
Cloudflare) take precedence over the wildcard; deleting such a record silently moves the name to the
|
||||
edge, where it only works if a connector serves it.
|
||||
|
||||
## Updates
|
||||
|
||||
Both are automatic:
|
||||
|
||||
- **OS:** Zincati stages new Fedora CoreOS releases (stable stream) and reboots only in the window
|
||||
03:00-04:00 UTC (`/etc/zincati/config.d/55-updates-strategy.toml`).
|
||||
- **sish:** upstream publishes only exact tags (`v2.24.0`), so `podman auto-update` can't follow a
|
||||
version line. `sish-update.timer` (daily ~05:00 UTC) runs `/usr/local/bin/sish-update`, which
|
||||
sets `Image=` in the quadlet to the newest tag matching `TRACK=v2.` (minor + patch, never a new
|
||||
major), restarts sish and rolls back if nothing listens on :5002 after 30s. `TRACK=v2.24.` limits
|
||||
it to patch releases.
|
||||
|
||||
```sh
|
||||
journalctl -u zincati -u sish-update
|
||||
systemctl start sish-update # check now
|
||||
```
|
||||
|
||||
Both restart sish (tunnels drop for a few seconds). A major sish release (`v3`) is a manual change
|
||||
of `TRACK` and `Image=`.
|
||||
|
||||
## First boot checklist
|
||||
|
||||
- `ss -tlnp`: sish on 22, 80, 443, 5002; sshd on 5001.
|
||||
- `journalctl -u sish`: `Loading ssh_host_ed25519_key as ssh-ed25519 host key` (the provisioned
|
||||
key, not a generated one).
|
||||
- After the first OS update: SSH on 5001 still works, `semodule -l | grep sshd_port_5001`.
|
||||
@@ -1,6 +0,0 @@
|
||||
# Source from anywhere: `. ./env.sh` (bash or zsh). Endpoint and node live in the talosconfig,
|
||||
# so plain `talosctl <cmd>` / `kubectl <cmd>` talk to cirrus-01.
|
||||
_cirrus_root=$(cd "$(dirname "${BASH_SOURCE[0]:-${(%):-%x}}")" && pwd)
|
||||
export TALOSCONFIG="$_cirrus_root/talos/talosconfig"
|
||||
export KUBECONFIG="$_cirrus_root/talos/kubeconfig" # created by: talosctl kubeconfig talos/kubeconfig
|
||||
unset _cirrus_root
|
||||
@@ -1,77 +0,0 @@
|
||||
# Talos: cirrus edge node
|
||||
|
||||
Single-node Talos control plane that runs only sish. Talos and Kubernetes are managed by hand
|
||||
with `talosctl`/`kubectl` (no Flux).
|
||||
|
||||
| File | |
|
||||
|---|---|
|
||||
| `controlplane.yaml` | Base config, **unmodified** output of `talosctl gen config` (gitignored, contains the cluster PKI) |
|
||||
| `worker.yaml` | Generated worker config, unused on a single node (gitignored) |
|
||||
| `talosconfig` | Admin client config for `talosctl` (gitignored) |
|
||||
| `patches/*.yaml` | Every change to the base config |
|
||||
|
||||
```sh
|
||||
export TALOSCONFIG=talos/talosconfig # run from the repo root
|
||||
N="-n 10.20.5.130 -e 10.20.5.130"
|
||||
```
|
||||
|
||||
## Base config + patches
|
||||
|
||||
The base file is never edited by hand; all changes live in `patches/`. The node's config is
|
||||
therefore always `controlplane.yaml` + `patches/*.yaml`, which keeps changes reviewable and lets
|
||||
a newly generated base (new node, new Talos defaults) get the same changes by reapplying the
|
||||
patches. `talosctl patch mc` merges a patch into the node's live config; it does not touch the
|
||||
local files.
|
||||
|
||||
| Patch | Purpose |
|
||||
|---|---|
|
||||
| `control-plane-scheduling.yaml` | Drops the control-plane `NoSchedule` taint so sish can run on the only node |
|
||||
| `unprivileged-ports.yaml` | `ip_unprivileged_port_start=22`: sish (non-root, hostNetwork) binds :22/:80/:443 |
|
||||
| `firewall.yaml` | Ingress firewall, default block (see below) |
|
||||
|
||||
Apply a single patch (dry run first; use `--mode try` for anything that can lock you out, it
|
||||
reverts automatically unless re-applied):
|
||||
|
||||
```sh
|
||||
talosctl $N patch mc --patch @talos/patches/<patch>.yaml --mode no-reboot --dry-run
|
||||
talosctl $N patch mc --patch @talos/patches/<patch>.yaml --mode no-reboot
|
||||
```
|
||||
|
||||
Check that the node matches the files (expect `No changes.`):
|
||||
|
||||
```sh
|
||||
talosctl machineconfig patch talos/controlplane.yaml \
|
||||
$(for p in talos/patches/*.yaml; do printf -- '--patch @%s ' "$p"; done) |
|
||||
talosctl $N apply-config --file /dev/stdin --dry-run
|
||||
```
|
||||
|
||||
## Firewall
|
||||
|
||||
Default action `block`. Loopback and replies to outgoing connections are always allowed.
|
||||
|
||||
| Open | Source | |
|
||||
|---|---|---|
|
||||
| 22, 80, 443, 2222, 20000-20099 tcp | anyone | sish (2222 = connector SSH, rest = forwards) |
|
||||
| 6443, 50000 tcp | anyone | Kubernetes API, Talos API (both client-cert authenticated) |
|
||||
| 53 udp/tcp | pod network `10.244.0.0/16` | CoreDNS forwards to the Talos host DNS |
|
||||
|
||||
Closed: flannel VXLAN 4789/udp, etcd 2379-2383, kubelet 10250, kube-proxy 10256, trustd 50001
|
||||
(open it to the node network only when a second node joins).
|
||||
|
||||
The sish ports must match `port-bind-range` in `kubernetes/base/sish/config.yml`. To add a raw
|
||||
TCP port outside 20000-20099, change both files together.
|
||||
|
||||
## New node
|
||||
|
||||
```sh
|
||||
talosctl gen config <cluster-name> https://<node-ip>:6443 --install-disk <disk> -o talos/
|
||||
talosctl machineconfig patch talos/controlplane.yaml \
|
||||
$(for p in talos/patches/*.yaml; do printf -- '--patch @%s ' "$p"; done) |
|
||||
talosctl apply-config --insecure -n <node-ip> --file /dev/stdin
|
||||
talosctl --talosconfig talos/talosconfig config endpoint <node-ip>
|
||||
talosctl --talosconfig talos/talosconfig -n <node-ip> bootstrap
|
||||
talosctl --talosconfig talos/talosconfig -n <node-ip> kubeconfig
|
||||
```
|
||||
|
||||
Back up `controlplane.yaml` and `talosconfig` outside this folder: they are the only copy of
|
||||
the cluster PKI and admin credentials.
|
||||
@@ -1,7 +0,0 @@
|
||||
# Single node: let workloads (sish) run on the control plane by dropping the
|
||||
# default control-plane NoSchedule taint.
|
||||
apiVersion: v1alpha1
|
||||
kind: KubeNodeConfig
|
||||
taints:
|
||||
node-role.kubernetes.io/control-plane:
|
||||
$patch: delete
|
||||
@@ -1,5 +0,0 @@
|
||||
# Single node: no cluster discovery, so no dependency on discovery.talos.dev.
|
||||
apiVersion: v1alpha1
|
||||
kind: DiscoveryServiceConfig
|
||||
name: default
|
||||
$patch: delete
|
||||
@@ -1,61 +0,0 @@
|
||||
# Ingress firewall: block everything that is not listed here. Loopback and
|
||||
# replies to outgoing connections are always allowed by Talos.
|
||||
#
|
||||
# Public TCP ports served by sish must match port-bind-range in
|
||||
# kubernetes/base/sish/config.yml (plus :80 HTTP and :2222 sish SSH).
|
||||
# Closed on purpose: flannel VXLAN 4789/udp (single node, unauthenticated),
|
||||
# etcd 2379-2383, kubelet 10250, kube-proxy 10256, trustd 50001 (only needed
|
||||
# when other nodes join).
|
||||
apiVersion: v1alpha1
|
||||
kind: NetworkDefaultActionConfig
|
||||
ingress: block
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: NetworkRuleConfig
|
||||
name: sish-public
|
||||
portSelector:
|
||||
ports:
|
||||
- 22 # gitea ssh (raw tcp forward)
|
||||
- 80 # sish http, routed by Host header
|
||||
- 443 # sish tls, routed by SNI
|
||||
- 2222 # sish ssh endpoint for connectors (public key auth)
|
||||
- 20000-20099 # reserved for raw tcp forwards
|
||||
protocol: tcp
|
||||
ingress:
|
||||
- subnet: 0.0.0.0/0
|
||||
- subnet: ::/0
|
||||
---
|
||||
# Talos API (apid) and Kubernetes API, both mTLS / client-cert authenticated
|
||||
apiVersion: v1alpha1
|
||||
kind: NetworkRuleConfig
|
||||
name: talos-and-kube-api
|
||||
portSelector:
|
||||
ports:
|
||||
- 50000
|
||||
- 6443
|
||||
protocol: tcp
|
||||
ingress:
|
||||
- subnet: 0.0.0.0/0
|
||||
- subnet: ::/0
|
||||
---
|
||||
# CoreDNS forwards to the Talos host DNS (forwardKubeDNSToHost), which pods reach
|
||||
# on the host, so the pod network needs DNS to the node
|
||||
apiVersion: v1alpha1
|
||||
kind: NetworkRuleConfig
|
||||
name: pod-dns
|
||||
portSelector:
|
||||
ports:
|
||||
- 53
|
||||
protocol: udp
|
||||
ingress:
|
||||
- subnet: 10.244.0.0/16
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: NetworkRuleConfig
|
||||
name: pod-dns-tcp
|
||||
portSelector:
|
||||
ports:
|
||||
- 53
|
||||
protocol: tcp
|
||||
ingress:
|
||||
- subnet: 10.244.0.0/16
|
||||
@@ -1,5 +0,0 @@
|
||||
# Static hostname instead of the generated auto: stable one.
|
||||
apiVersion: v1alpha1
|
||||
kind: HostnameConfig
|
||||
auto: off
|
||||
hostname: cirrus-01
|
||||
@@ -1,6 +0,0 @@
|
||||
# Lets non-root processes bind ports >= 22, so sish (hostNetwork, uid 65534,
|
||||
# no capabilities) can listen on :22 (Gitea SSH), :80 and :443.
|
||||
# The edge runs nothing else that could grab 22-79.
|
||||
machine:
|
||||
sysctls:
|
||||
net.ipv4.ip_unprivileged_port_start: "22"
|
||||
Reference in New Issue
Block a user