secure-boot-cert-rotation

# secure-boot-cert-rotation

Triage and fix the **Microsoft Secure Boot 2011→2023 certificate rotation** on a real, mixed fleet.
Microsoft's original 2011 Secure Boot CAs expire in 2026; their 2023 replacements take over. This skill
exists to stop the two failure modes operators actually hit: (1) **panic** — believing servers will stop
booting on the expiry date (they won't), and (2) **applying the wrong tool** — reaching for `fwupd`/LVFS on
hardware and VMs it cannot serve, instead of the firmware-native path.

The work is almost never the cert write itself — it's **knowing which of three firmware surfaces a given
machine has** (Dell host firmware, generic Linux host firmware, or a VM's virtual OVMF varstore), because each
is updated by a different mechanism and `fwupd` only covers one of them.

## The one load-bearing fact — read this before anything else

**UEFI firmware does not check a certificate's expiry date when validating Secure Boot signatures.** EDK2 sets
`NO_CHECK_TIME` on PKCS#7 verification on purpose (there is no trustworthy clock at boot; enforcing `notAfter`
would be a self-inflicted brick vector). Canonical, Red Hat, fwupd, and LWN all state this independently.

Consequences, and the entire reason this is a *hygiene* task and not a *fire*:

- **Every machine that boots today keeps booting after the expiry dates.** Nothing breaks on the deadline.
- The **real** risks are forward-looking:
  1. **Forward-compat** — once a distro ships a shim/bootloader signed *only* by the **Microsoft UEFI CA 2023**,
     a machine whose firmware `db` lacks that 2023 cert can't validate the new binary. On Linux this surfaces
     as "package-management failures / can't install updates"; on new installs/PXE as a failed deploy.
  2. **Revocation freeze** — a machine stuck on the expired **2011 KEK** can't receive new `dbx` revocations, so
     known-vulnerable bootloaders stay trusted. (Lower on Linux, which revokes via **SBAT**, not `dbx`.)
- **So the fix everywhere is: get the Microsoft 2023 `db` cert into the firmware/varstore** (and the 2023 KEK
  where the platform offers a path). Mechanism + dates + the 2011→2023 map: `references/mechanism.md`.

## Routing — pick the surface, then the reference

A machine has exactly one of three firmware surfaces. Identify it first; everything else follows.

| Surface | Who has it | Cleanest fix | Reference |
|---|---|---|---|
| **Dell PowerEdge host firmware** | Any Dell bare-metal node (incl. Linux servers and Harvester *hosts* on Dell) | Recent BIOS already carries 2023 keys, **staged → applied on a controlled reboot** (Lifecycle Controller pass). Audit + reboot; `ResetAllKeysToDefault` via Redfish on a new-enough BIOS; Ansible for the fleet. | `references/dell-poweredge.md` |
| **Generic Linux host firmware** | Non-Dell bare metal, or OS-side enrollment on any host | Manual `db` append of Microsoft's signed 2023 payload — **no fwupd, no Setup Mode** (it self-auths via the 2011 KEK). | `references/linux-bare-metal.md` |
| **VM virtual OVMF varstore** | Harvester / KubeVirt (and any QEMU/OVMF) guest VMs | iDRAC does **not** touch this. Governed by the **Harvester version → virt-launcher → OVMF** (floor **v1.6.0**); then ephemeral-vs-persistent-NVRAM triage. | `references/harvester-vms.md` |

`fwupd`/LVFS is the desktop/laptop path. It is the **wrong tool** for Dell PowerEdge (firmware ships via
iDRAC/DSU, not LVFS), for VMs (no capsule path), and on stock Ubuntu it is also simply **too old** (22.04 ships
1.7.9, 24.04 ships 1.9.34; the `uefi-db`/`uefi-kek` cert plugins need **fwupd ≥ 2.0.8**). If `fwupdmgr` "does
nothing" on a server, that is expected — use the firmware-native path. Details in each platform reference.

## Workflow

### 1. Audit before you touch anything (read-only)

Always start by reading what's actually enrolled — the fix and its urgency depend on it. Verdict: *NEEDS
UPDATE* if `db` has the 2011 Microsoft UEFI CA but not `Microsoft UEFI CA 2023`; *GOOD* if the 2023 certs are
present.

```bash
# Linux host or in-guest:
sudo mokutil --sb-state                                              # Secure Boot on?
sudo mokutil --kek | grep -i "KEK 2K CA 2023"                       # 2023 KEK present?
sudo mokutil --db  | grep -iE "Microsoft UEFI CA 2023|Windows UEFI CA 2023"
sudo efi-readvar -v db | grep -i 2023                               # efitools alternative
```
Dell out-of-band: `racadm ... bioscert view --all` or Redfish `GET /redfish/v1/Systems/System.Embedded.1/SecureBoot/SecureBootDatabases`.
Harvester VMs: first find which VMs even *use* Secure Boot (`kubectl ... efi.secureBoot`) — many don't, and
those are moot. Exact commands per reference.

### 2. Establish the surface and route (table above), then apply that reference's runbook

Each reference is self-contained: audit → cleanest fix → fallback → verify.

### 3. Verify after, on a sample, before fleet rollout

Re-run the §1 audit. **Test one host per model+firmware (or one VM per template) before the fleet** — failure
modes cluster by firmware version (HP/Fujitsu block standalone db updates; old BIOSes exhaust NVRAM). Cross-
cutting gotchas, ordering, and the do-nothing risk timeline: `references/gotchas-and-decisions.md`.

## House rules (the hard-won ones — encode these into every answer)

1. **Don't fearmonger.** Lead with "nothing stops booting on the deadline." Expiry ≠ revocation; firmware
   ignores `notAfter`. The honest framing is forward-compat + revocation-freeze on a slow fuse, not a brick
   event. Getting this wrong sends operators into needless emergency reinstalls.
2. **`db`-first works without the 2023 KEK.** The Microsoft 2023 `db` payloads are signed by the *old 2011 KEK*
   that every machine already has, so they self-authenticate and append with `efi-updatevar -a`/`sbkeysync` —
   no fwupd, no Setup Mode. There is **no generic Microsoft-signed 2023 KEK payload** (KEK updates are per-OEM,
   PK-signed); the 2023 KEK arrives via OEM BIOS (Dell), Windows servicing, fwupd's per-hardware LVFS KEK, or
   `virt-fw-vars` for VMs. Prioritize the db cert — it is what boots future 2023-signed bootloaders.
3. **Reinstalling the OS does NOT fix firmware.** `db`/`KEK` live in firmware NVRAM (or a VM's varstore),
   independent of the OS. A 26.04 reinstall only ships a 2023-aware shim/grub — the cert still needs enrolling.
   Decouple "patch in place vs reinstall" from "fix the certs". `references/linux-bare-metal.md` § Reinstall.
4. **Verify version claims against the REAL artifact, never upstream commit dates.** "Does package/image X carry
   the 2023 certs?" is answered by reading the *published* artifact — distro changelog/advisory, or the bytes of
   the actual varstore/ISO/RPM — because distros **backport** certs onto old bases (Red Hat and SUSE both did).
   Inferring from an upstream edk2 tag date gives a confidently-wrong answer. (This is exactly how the Harvester
   floor was pinned to **v1.6.0** by `openssl`-parsing the varstore in each release's `virt-launcher`, not by
   the Oct-2025 upstream commit.) `references/harvester-vms.md` shows the proof method.
5. **Ground volatile numbers; don't assert from memory.** Dell per-generation BIOS minimums, Ubuntu fwupd
   pocket versions, and Harvester release dates change. State a specific version only if it's machine-reported,
   freshly grounded (Dell KB / Launchpad / `gh release`), or marked UNVERIFIED. The methodology (mechanism,
   trust chain, surfaces) is what this skill asserts; the leaf numbers are grounded per use.
6. **Right tool per surface.** Don't recommend `fwupd` for Dell PowerEdge or VMs, and don't recommend `rpm -q`
   on a Harvester *node* to check guest OVMF (it isn't there — guest OVMF ships in the `virt-launcher`
   container). Match the mechanism to the firmware surface (routing table).
7. **Audit → sample → fleet.** Read what's enrolled before acting; pilot one host/VM per model+firmware before
   rolling out. Transient unbootability is the real (recoverable) risk, not hardware damage.

## References

- `references/mechanism.md` — what expires + exact dates, the 2011→2023 cert map (KEK vs db), why firmware
  ignores expiry, the PK→KEK→db trust chain, the no-generic-2023-KEK fact, SBAT-vs-dbx on Linux.
- `references/dell-poweredge.md` — iDRAC9 path: per-generation BIOS minimums, staged-until-reboot mechanism,
  `ResetAllKeysToDefault` via Redfish/racadm, `bioscert` audit, the `dellemc.openmanage` Ansible module.
- `references/linux-bare-metal.md` — audit commands, the fwupd-free manual `db` append (Microsoft signed-payload
  filenames + URLs), firmware-menu (Key Management) enrollment from raw `PreSignedObjects/*.der` + the ESP
  staging trick (`/boot/efi`, no USB/virtual-media), the fwupd-snap fallback + Ubuntu pocket reality, the
  patch-vs-reinstall decision.
- `references/harvester-vms.md` — the two layers (Dell host vs guest OVMF), guest OVMF ships in `virt-launcher`
  (floor **v1.6.0**, per-line table, the artifact-proof method), host Secure Boot + bug #7343, the
  ephemeral-vs-persistent NVRAM triage with exact `kubectl` commands, `virt-fw-vars` injection.
- `references/gotchas-and-decisions.md` — order of operations (PK→KEK→db), HP/Fujitsu standalone-db block,
  NVRAM exhaustion, BitLocker/PCR7 reseal, the do-nothing risk timeline, the disable-Secure-Boot tradeoff.
- `references/sources.md` — primary sources (Microsoft, Dell, Red Hat, Canonical, fwupd/LVFS, SUSE/Harvester,
  KubeVirt) with one-line credibility notes.