seaworld008/centos7-almalinux-upgrade

CentOS 7 → AlmaLinux 10 In-Place Upgrade

This is a workflow distilled from a real production migration (Huawei Cloud bastion, CentOS 7.7 → AlmaLinux 10.2, nginx/Docker/xxl-job/MySQL/NetBird/kubectl all preserved). The goal is to safely, rollback-ably, and verifiably upgrade each machine, while also being able to push through a fleet.

Core idea: Assess first → Snapshot for safety → Small, step-by-step hops (Pre-check → Upgrade → Reboot → Compare Baseline → Cleanup) → Finally modernize. Almost all pitfalls come from 'non-standard' things (self-compiled programs, pip packages, third-party repos, self-packaged openssh, container restart policies, VPN access paths), so the discovery phase is the most valuable.

When NOT to In-Place Upgrade

First, check the survey.sh output for BLOCKERs. Consider creating a new AlmaLinux 10 machine and migrating the service instead of an in-place upgrade in these cases:

• It is a Kubernetes node (kubelet is running) — treat it as a cluster-level change; follow references/kubernetes.md (kubeadm 1.21 + dockershim + flannel/ipvs tested on EL10, ~6 hours for a 3m3w cluster); or drain and replace the node.
• The host has RPM-installed databases needing cross-major-version upgrades (e.g., MariaDB 5.5→10.11, PostgreSQL 9.2→16) with large data volumes.
• Non-x86_64 architecture, btrfs filesystem, or the system disk is extremely small and cannot be expanded.
• The CPU does not support x86-64-v3 (AlmaLinux 10 standard edition won't run; consider upgrading only to 9, or using the AlmaLinux 10 x86_64_v2 variant).
• The services are already containerized, with configuration in compose/k8s — creating a new machine and redeploying is often faster than an in-place upgrade.

When unsure, present the reasoning to the user and let them decide.

Toolkit

All scripts are located in the skill directory scripts/ (uploaded via SSH from the local machine to the target /root/os-upgrade-kit/ for execution):

Script	Purpose
remote.py	Runs locally: reads credentials from an asset spreadsheet (xlsx/csv), executes commands via SSH, uploads/downloads files, waits for reboot; keepalive enabled; does not print passwords
survey.sh	Read-only discovery; outputs graded RISK FLAGS at the end (BLOCKER/HIGH/MEDIUM/INFO)
backup.sh	Backs up configurations + generates a business baseline baseline.tsv (services/ports/containers/per-domain HTTP status codes/kubectl)
state.sh / verify.sh	Generates current state / compares against baseline, outputs FAIL/WARN
prep-el7.sh	EL7 preparation: repos (fix repos), update to 7.9, pipfix, inhibitors
convert-el8.sh	When the source system is CentOS 8: prep → run (official almalinux-deploy) → post, converting to AlmaLinux 8
swap-openssh.sh	Replaces a self-compiled openssh with the official package (with automatic fallback)
hop.sh	Each hop: prep→pre→report/answer→up→reboot→post
nginx-static-build.sh	Static rebuild of a self-compiled nginx at the same version (usable throughout the transition)
fleet_summary.py	Runs locally: aggregates survey results from multiple machines → classifies them into A/B/C/D, marks VPN/EL8, gives canary and batch recommendations (--groups to ensure primary/secondary of the same business are in different batches)
finalize.sh	EL10 cleanup: wrap compose, align kubectl with cluster, helm, postfix lmdb, nginx.org repo, cgroup v1 compatibility, SysV wrapping, legacy iptables modules, redis→valkey, convert manually started processes to systemd services
k8s.sh	Kubernetes node: backup (etcd snapshot/config/binaries/pod list, automatically pin versions), stop, nfs-off/nfs-on, nm, fix, check, pods (compare with pre-upgrade), jvm-maxram (OOM from older JDK under cgroup v2)

When running locally in Windows Git Bash, remote.py automatically handles MSYS's rewriting of /root/... to C:/Program Files/Git/root/....

Common usage:

export FLEET_INVENTORY="/path/to/servers.xlsx"   # Columns: name, host, port, username, password (contains password, DO NOT commit to repo)
S=<skill directory>/scripts
python $S/remote.py HOST put $S /root/os-upgrade-kit
python $S/remote.py HOST run "bash /root/os-upgrade-kit/survey.sh > /root/os-upgrade-kit/survey.txt; sed -n '/RISK FLAGS/,\$p' /root/os-upgrade-kit/survey.txt"
python $S/remote.py HOST wait-ssh --timeout 3600

Process

Phase 0: Things to Confirm with the User Before Starting (All Mandatory)

These are "the user's decisions", do not assume for them:

1. Cloud snapshot/full machine backup is done (system disk + data disks). This is the only reliable rollback method; Claude cannot do this inside the machine.
2. Maintenance window: Each hop requires a reboot, the entire process takes about 2-4 hours (mostly downloading packages, depending on mirror speed); the service will be interrupted multiple times.
3. Out-of-band console is available: Alibaba Cloud VNC / Huawei Cloud VNC / AWS EC2 Serial Console (prerequisites vary by provider, see references/cloud-providers.md). If the upgrade gets stuck in initramfs, this is the only way to recover.
4. Access path: How do you (local machine) connect to this server? If it's via VPN/tunnel and the VPN runs on this machine (the survey will report a BLOCKER), be very careful with reboots and container operations — our real incident was docker stoping all containers, which caused the NetBird routing container not to start on boot, losing connectivity to the entire VPC.
5. What is already unused: After discovery, provide the list to the user and let them confirm which components/sites can be decommissioned (e.g., PHP, old sites). Explain the impact before deleting (e.g., "removing PHP will cause xxx site to 404"), and let the user make the final call.

Phase 1: Discovery + Backup + Baseline

1. Upload the toolkit, run survey.sh, review each RISK FLAG line by line, and use references/app-compat.md to determine a treatment plan for each.
2. Run backup.sh (defaults to a non-system disk like /data), note the BACKUP_DIR. The baseline.tsv is the reference for comparison after each hop — items that were already abnormal before the upgrade (e.g., 502) should be noted in the report as "already existed before the upgrade" to avoid confusion later.
3. Present the discovery conclusions, treatment plans, and items requiring user decisions to the user all at once (see Phase 0).

Phase 2: EL7 Preparation

bash prep-el7.sh repos <MIRROR>      # Domestic recommended: https://mirrors.aliyun.com; Alibaba Cloud internal: http://mirrors.cloud.aliyuncs.com; Overseas: upstream
bash prep-el7.sh update              # Update to 7.9; packages showing Failed are mostly pip conflicts
bash prep-el7.sh pipfix
bash prep-el7.sh inhibitors
bash swap-openssh.sh                 # Only if survey reports non-official openssh; verify with a [new connection] after completion

Self-compiled nginx and other programs that depend on EL7 libraries: Use nginx-static-build.sh to rebuild statically on EL7 (or accept that they will be disabled during the transition, and finally install the official package). Host databases: Perform a logical backup first.

When the source system is CentOS Linux 8 / CentOS Stream 8 (the survey will report a BLOCKER): ELevate does not support upgrading from it. Skip the EL7 preparation above; first perform an in-place conversion, then start from phase 3's 8→9 hop:

bash convert-el8.sh prep <MIRROR>                    # Fix repos (vault, EPEL8), pip conflicts, redundant kernel-devel
bash convert-el8.sh update                           # Upgrade to 8.5 (almalinux-deploy requires ≥8.4)
bash convert-el8.sh run  <MIRROR> <almalinux-deploy.sh>   # Official almalinux-deploy: -d allows downgrades, -l uses mirror
bash hop.sh reboot && python remote.py HOST wait-ssh   # Reboot into AlmaLinux kernel
bash convert-el8.sh post                             # Remove non-AlmaLinux kernels, dnf upgrade

Domestic servers usually cannot access GitHub raw; download almalinux-deploy.sh locally first, then upload it.

Phase 3: Each Hop (7→8, 8→9, 9→10 — each performed once)

bash hop.sh prep <MIRROR>     # Install ELevate for this hop, target repos, switch docker-ce/epel vendor repos to mirrors, disable test/nightly sections, pip cleanup, probe reachability
bash hop.sh pre               # Run preupgrade in background and wait → automatic report
bash hop.sh answer            # Automatically answer known security questions; for other blockers, process using the fix commands in the report, then run pre again, until 0 blockers remain
bash hop.sh report            # Focus on "packages that will be removed" and "destination of critical packages" — if any business depends on them, find a way first
bash hop.sh up                # Download packages + generate upgrade environment in background
bash hop.sh reboot [additional container names to stop]   # Only stop stateful containers with restart=always; do not touch VPN containers and unless-stopped containers; if the business says "don't touch", use --no-stop
python remote.py HOST wait-ssh --timeout 3600   # Wait locally; timeout → use out-of-band console, see references/recovery.md
bash hop.sh post <MIRROR>     # Check for failure markers, clean leapp/old kernels/residual repos, dnf upgrade
bash verify.sh <BACKUP_DIR>   # FAIL must be handled; WARN: determine if expected

Fix business issues after each hop before proceeding to the next hop; do not accumulate fixes until the end. For blockers and treatments specific to each hop, see references/pitfalls.md's "By Hop" section.

Long tasks always run in the background: The scripts use systemd-run for leapp/dnf, so they are not affected if the SSH session drops; you just poll. When writing your own commands, follow the same principle — do not let an SSH session hang for tens of minutes.

Phase 4: EL10 Cleanup & Modernization

Choose finalize.sh subcommands as needed, and consult references/app-compat.md:

• Self-compiled nginx → nginx.org official stable package (change ssl on; to listen 443 ssl;, TLS1.2/1.3, certificates in /etc/nginx/ssl, systemd autostart)
• docker-compose v1 → wrap into compose v2; add shim for cgroup v1 legacy applications
• kubectl → pkgs.k8s.io repo, align minor version with the cluster; helm → latest v3
• postfix hash → lmdb; convert startup items in rc.local to systemd unit files
• dnf -y upgrade full update, confirm dnf check-update returns empty
• Finally, reboot once more to test autostart, run verify.sh; all services must start on their own for the upgrade to be considered complete

Phase 5: Delivery Report

Write a report to the user using the structure from assets/report-template.md: version comparison, business comparison table (before/after upgrade), compatibility changes made, outstanding items and tasks for the user (e.g., certificate expiry, unrepaired cloud vendor plugins), backup location. For fleet scenarios, also update the asset tracking table (assets/fleet-tracker.csv).

Phase 6: Accumulate Experience (Give Back to the Skill Repository)

This skill evolves through real execution; project URL: https://github.com/seaworld008/centos7-lighthouse . After each real execution (success, failure, or rollback — all count):

• Write a sanitized field report following the repository's field-reports/TEMPLATE.md (do not include real IPs, domain names, hostnames, or credentials).
• Add new issues to references/pitfalls.md; if automatable, add detection to survey.sh and automatic handling improvements to the scripts.
• If the current working directory is already this repository, follow its AGENTS.md; otherwise, give the report to the user and suggest submitting an issue or PR.

Key Principles (Each Corresponds to a Real Incident)

• Never docker stop $(docker ps -q) before reboot: Containers with unless-stopped that are manually stopped will not auto-start; if a VPN container is stopped, you will lose connectivity. Use hop.sh reboot.
• Anything installed into system site-packages via pip must be moved before each hop: If they conflict with RPM files, the upgrade will fail mid-way in initramfs, leaving the system in emergency mode in a "half-upgraded" state.
• Unknown, non-official items must all be handled first: Self-compiled openssh, self-compiled nginx, third-party repos, mysql-community-libs replacing mariadb-libs… leapp does not handle them, and they will break after the upgrade.
• Use domestic mirrors in China, and disable vendor repos' test/nightly sections: mirrorlist/metalink are often slow or time out; a timeout from the docker nightly repo can directly cause leapp to fail. Huawei Cloud mirrors may rate-limit with 429; Alibaba Cloud is the most stable.
• pkill -f / pgrep -f patterns must not appear in your own command lines, otherwise they will kill your own SSH session (we fell for this twice). Use PIDs, script files, or patterns like [l]eapp.
• Compare against baseline after each hop: When a business issue is found during a hop, fix it in that hop — e.g., cgroup v2 causing Yearning to crash, docker-compose v1 becoming invalid.
• Decisions that change business behavior belong to the user: Deleting components, decommissioning sites, disabling TLS1.0/1.1, switching database major versions.

Reference Documents (Read as Needed)

• references/pitfalls.md — All pitfalls: symptom → cause → treatment, categorized by hop + general. Look here first if you encounter an error.
• references/app-compat.md — Compatibility handling for various applications across EL7→EL10: nginx, Docker/compose/cgroup v2, Java, Python, PHP, databases, postfix, network, SSH, kubectl/helm, SysV, TLS encryption policies.
• references/cloud-providers.md — Alibaba Cloud/Huawei Cloud/AWS differences: snapshots, out-of-band console prerequisites, mirror sources, cloud vendor agents, cloud-init.
• references/recovery.md — How to recover when stuck in initramfs emergency mode, loss of connectivity on boot, or a half-upgraded state (includes VNC operation tips).
• references/fleet.md — Rolling out to dozens or hundreds of machines in batch: asset spreadsheet, parallel discovery, classification, canary, batching, record keeping.
• references/kubernetes.md — In-place upgrade of K8s nodes: things to clarify before starting (access path/control plane ingress/etcd/storage), compatibility conclusions, recommended sequence, pitfalls encountered, recovery checklist.