harvester-upgrade

# harvester-upgrade

Plan, sequence, and de-risk a **community-edition Harvester HCI** upgrade across a real fleet — typically off an
**EOL line** (1.5.x) up to the latest *settled* stable. The hard part is almost never the ISO/UI upgrade click.
It is the **coordination**: Harvester is one bundle whose embedded RKE2, KubeVirt, Longhorn, SLE Micro, and
Rancher pairing all move together, the upgrade controller **picks its own node order**, and the workloads that
matter most — VM-hosted Kubernetes control planes — must stay quorate while each host reboots underneath them.

**Community editions only.** SUSE sells "SUSE Virtualization" (Prime) as a paid *support subscription on the
same bits* — it is **not** a different artifact set, and **patch releases are community-downloadable** (House
Rule #1). Flag Prime-only support claims; never build a plan on them.

## Companion to k8s-components-checker and rancher-upgrade — who owns what

This skill is the Harvester-side upgrade-methodology companion in the same trio (`rancher-upgrade` SKILL.md
explicitly defers Harvester host→guest coordination here). Respect the boundary to avoid drift:

| Question | Skill |
|----------|-------|
| Harvester↔RKE2↔Rancher *pairing numbers*, Node-Driver ranges, per-minor compat | **k8s-components-checker** — `references/compat/harvester.md` is the source of truth. **Cite it; don't restate.** |
| Is a component (Cilium, Rook, …) OK on k8s 1.NN on one cluster? | **k8s-components-checker** (per-cluster verdict) |
| How to upgrade the **Rancher server** itself (the external Rancher's own 2.11→2.14 walk, KDM, CAPI/Turtles) | **rancher-upgrade** |
| How to plan/sequence/execute the **Harvester** upgrade + keep guest control planes alive | **this skill** |

## The load-bearing facts — read these before anything else

1. **No minor skipping, ever.** The only supported path is one minor at a time: `1.5.x → 1.6.x → 1.7.x →
   1.8.x` (each Harvester minor bumps embedded RKE2 exactly one k8s minor; skipping a k8s minor is unsupported
   upstream). Intermediate **patches** within a jump *may* be skipped (`1.5.2 → 1.6.1` directly) — land on each
   minor's latest patch. **Never hand-edit embedded RKE2** — it is locked to the Harvester version.
2. **External Rancher leads every hop.** When Harvester is imported into an external Rancher, each hop is a
   three-step sequence in order: **upgrade Rancher → upgrade the Harvester UI extension → upgrade Harvester.**
   The pairing is required (1.6↔Rancher 2.12, 1.7↔2.13, 1.8↔2.14); a mismatch yields "VM tab missing" /
   unmanageable cluster. The external Rancher's own multi-minor walk is its own gated project — defer to
   `rancher-upgrade`. `references/external-rancher-coupling.md`.
3. **The node-upgrade order is not operator-choosable.** Order is delegated to RKE2/Rancher and forced
   strictly serial (concurrency=1, one-host-at-a-time interlock). The only "pause between nodes" knob
   (`nodeUpgradeOption: manual`) **exists only from v1.7.0+** — unavailable for the 1.5→1.6 and 1.6→1.7 hops.
   So protect VM-hosted control planes **structurally**, not with a button. `references/controlled-flow-and-node-order.md`.
4. **Spreading the control-plane VMs is the real safety mechanism.** Anti-affinity so the etcd VMs land on
   distinct hosts → a one-host-at-a-time upgrade only ever touches **one** member, and quorum holds whether that
   member migrates or is cleanly stopped. **Do not assume a busy etcd VM will live-migrate** — write-heavy VMs
   often *never converge* in pre-copy (Harvester ships auto-converge off and a self-throttling `bandwidthPerMigration:
   0`), so for production etcd plan to take one member down **cleanly** per host and let quorum cover it, rather
   than relying on live migration. `references/guest-rke2-survivability.md`.
5. **There is no downgrade.** A bricked host = rebuild + restore. Back up **VMs to NFS/S3** (not in-cluster
   snapshots) **and** take a **guest-cluster etcd snapshot** before each hop. Volume health (single-replica /
   degraded Longhorn) is the #1 thing that blocks a node drain — heal it first.
   `references/landmines-and-rollback.md`.

## Workflow

### 1. Establish the change set (detect, don't assume)
Read the running versions on the (air-gapped) cluster, then map onto the ladder. Never assume the starting
patch. Commands + version→minor map: `references/version-ladder-and-detection.md`.

### 2. Pick the target — latest *settled* stable, not bleeding-edge
"Latest stable" usually means the newest minor **that has a patch released** and a settled Rancher pair — not a
fresh `.0`. Avoid a first-of-minor release for a controlled upgrade off EOL hardware; landing one minor lower
often also avoids the newest Rancher minor's churn. Ground the actual latest patch of each minor via `gh`
(House Rule #2) and apply look-ahead (House Rule #5). `references/version-ladder-and-detection.md`.

### 3. For each hop: external Rancher → UI-extension → pre-flight → Harvester → verify guests
Each hop is self-contained: upgrade the external Rancher + UI extension to the pair, run the **enforced
pre-flight gates**, then the Harvester upgrade, then verify the guest control planes before the next hop.
Per-hop breaking changes + ordered runbook: `references/per-hop-runbook.md`. Pre-flight gate list: § below.

### 4. Control the flow and keep guests alive
Set up the structural protection (anti-affinity spread, N+1, MAC-pinned IPs, migratable VMs) BEFORE starting;
use the v1.7.0+ pause-map only where available. `references/controlled-flow-and-node-order.md` +
`references/guest-rke2-survivability.md`.

## Pre-flight gates (run before EVERY hop)

Harvester **blocks** an upgrade that fails these. Run the official pre-check (`bash pre-check/v1.x/check.sh -v`
from `harvester/upgrade-helpers`) on a control-plane node and clear every failure. The high-value gates:

- **Free system space** ≥ 30 GiB on `/usr/local` **and** projected post-image-load usage < 85% (kubelet
  imageGC). Fix with `crictl rmi --prune`.
- **All Longhorn volumes `healthy`** — no degraded/faulted, **no single-replica volumes** (they block drain),
  no stale attachments.
- **All nodes Ready** (none cordoned); **CAPI `local` Provisioned**, machine-count == node-count, all `Running`.
- **Certs > 7 days** from expiry; **NTP in sync**; **pods Ready** (rancher-logging healthy, Fleet bundles ok).
- **Back up first** (VM Backup to NFS/S3 + guest etcd snapshot); **disable recurring Longhorn jobs**.

Full gate table with thresholds + sources: `references/landmines-and-rollback.md` § Pre-flight.

## House rules (encode into every plan)

1. **Community editions only.** Patch releases (1.6.1, 1.7.1, …) ARE community — verify against the real
   artifact (public ISO + the official 2-month patch cadence), not a "x.y.0=community / x.y.1+=Prime" myth.
   "Prime" = paid support on identical bits. `references/version-ladder-and-detection.md` § Editions.
2. **Never invent versions; ground or abstain.** The ladder/pairing *mechanics* are this skill's methodology;
   specific latest-patch numbers, GA dates, and "fixed in vX" claims are volatile — state them only if
   cluster-reported, freshly `gh`-grounded (anti-confirmation: enumerate tags, derive per-minor latest, never
   ask "does vX exist"), or marked UNVERIFIED. Run `gh` with valid auth (enumeration exhausts anonymous).
3. **Rancher leads, no minor skips** (both axes; fact 2). Cite `rancher-upgrade` for the Rancher chain — don't
   re-derive it.
4. **Structural safety over the pause button** (facts 3–4): the pause knob is v1.7.0+ and field-flaky, so design
   in anti-affinity spread + serial interlock + MAC-pinned IPs (and N+1 where possible) before the first host.
5. **Look-ahead version targeting.** Recommend the target that also covers the *next* planned hop, not the bare
   immediate minimum — and don't land on a fresh `.0`/a minor at its support ceiling. (Same rule as
   k8s-components-checker House Rule #9 / rancher-upgrade House Rule #4.)
6. **No downgrade — back up before every hop** (fact 5): VM Backup to NFS/S3 (filesystem-consistent via
   qemu-guest-agent) + a guest etcd snapshot; in-cluster VM snapshots are NOT DR.
7. **Cite the real source per claim.** Pairing/Node-Driver numbers → `compat/harvester.md`; Rancher-side →
   `rancher-upgrade`; everything else → the reference file that carries the doc/issue evidence.

## References

- `references/version-ladder-and-detection.md` — the no-skip ladder + component table, the community-vs-Prime
  edition reality (corrected), choosing the target, and air-gapped version-detection commands (Harvester /
  RKE2 / KubeVirt / Longhorn / UI-extension).
- `references/external-rancher-coupling.md` — the Rancher→UI-extension→Harvester order, the required pairing
  table, air-gapped UI-extension install (`ui-plugin-catalog` image map), mismatch failure modes, and how the
  external Rancher's own chain gates the campaign (pointer to `rancher-upgrade`).
- `references/controlled-flow-and-node-order.md` — node order isn't operator-choosable; what the upgrade gates
  on between hosts (Longhorn yes, guest-etcd no); the 3-host physics; the per-version control table (`restoreVM`
  truth, pause-map is 1.7.0+); the guaranteed no-outage pause-map procedure; the **native PDB + VMI-readinessProbe
  gate** for 1.5/1.6 (with the pinning-bypass + must-test caveats); why the 1.8 upgrade-manager is off-limits.
- `references/guest-rke2-survivability.md` — making self-managed RKE2 guests Harvester-aware and survivable:
  anti-affinity spread (the #1 lever), N+1 capacity, live-migration prerequisites, qemu-guest-agent, the
  Harvester cloud-provider (CCM) + CSI standalone install, etcd-during-migration reality, MAC-pinned IPs.
- `references/per-hop-runbook.md` — per-hop (1.5→1.6, 1.6→1.7, 1.7→1.8) breaking changes, manual pre-steps, and
  the ordered runbook (wicked→NetworkManager, Intel NIC rename, DHCP client-ID, secondary-VLAN strip, etc.).
- `references/landmines-and-rollback.md` — the enforced pre-flight gate table, the landmine quick-reference
  (with issue numbers), Longhorn-as-#1-blocker, and the no-downgrade backup/rollback reality.
- `references/sources.md` — primary sources (Harvester docs, release notes, controller source, upgrade-helpers,
  CSI/CCM) with a freshness ledger.
</content>