#Node Preprocessing

Before installing the global cluster, all nodes (control plane nodes and worker nodes) must complete preprocessing.

#Supported OS and Kernel Versions

The following table lists the supported operating systems, their validated versions, and the corresponding tested kernel versions.

The platform enforces the version-matching granularity declared in the support list:

• OS version: Use the distribution release listed below. A different major or minor release is not officially supported unless it is also listed.
• Kernel version: Use an official kernel supplied by the operating system vendor. When the list gives an exact x.y.z-build value, the build suffix can vary but x.y.z must match. When the list names an operating system release without a kernel value, use the official kernel shipped with that release. Do not assume that an arbitrary newer kernel is supported.

#x86

#ARM

#Operating System Support Scope Changes

4.4 introduced Kubernetes 1.35, which removed cgroup v1. Nodes in a cluster running Kubernetes 1.35 or later must use cgroup v2 and Linux kernel 5.8 or later.

Because their kernels are earlier than 5.8, the following operating systems were supported through 4.3 and are no longer in the support scope from 4.4 onward:

• Kylin Linux Advanced Server V10 and V10 SP3 (kernel 4.19)
• Ubuntu 20.04 LTS (kernel 5.4)
• RHEL 7.x and RHEL 8.x (kernel 3.10 and kernel 4.18)
• CentOS 7.x (kernel 3.10)

If your nodes run one of these operating systems, choose one of the following paths. Moving to a newer release from the same operating system vendor is usually the cheapest option, because it keeps your existing vendor relationship, tooling, and operational practices:

• Move RHEL 7.x or RHEL 8.x nodes to RHEL 9.6.
• Move Ubuntu 20.04 LTS nodes to Ubuntu 22.04 LTS.
• Move Kylin Linux Advanced Server V10 or V10 SP3 nodes to Kylin Linux Advanced Server V11 2503.
• Move the nodes to Alauda OS. See the Immutable Infrastructure documentation. Alauda OS is an immutable operating system, so this path changes how the nodes are managed and is usually the most expensive option.
• Stay on 4.3.x until you can complete the operating system change.

CentOS 7.x has no successor in the support list. Move those nodes to one of the supported operating systems above.

#Execute the Quick Configuration Script

The installation package provides a script for quickly configuring nodes.

Unzip the installation package to obtain the init.sh script file in the res directory. Copy the script file to the nodes and ensure that you have root privileges.

Execute the script:

bash init.sh

#Node Checks

The following lists all the checks that must be completed on the nodes. Depending on the node's role, the required checks will vary. For example, some checks apply only to control plane nodes.

Checks are divided into two categories:

• ✅ Indicates a check that must pass.
• ⚠️ Indicates a check that must be met in specific scenarios. Please determine whether the corresponding conditions are met according to the instructions. If they are, you must resolve them.

The following is the list of checks:

• OS and Kernel

  • ✅ The machine's grub boot configuration must have the transparent_hugepage=never parameter.
  • ✅ Check whether the kernel modules ip_vs, ip_vs_rr, ip_vs_wrr, and ip_vs_sh are enabled.
  • ⚠️ If the global cluster plans to use Kube-OVN CNI, the kernel modules geneve and openvswitch must be enabled.
  • ✅ SELinux, AppArmor, and the host firewall must be disabled, unless your security policy requires one of them to stay enabled. In that case the node must be configured as described in Hardened Node Security Configuration before you install the platform.
    • init.sh disables all three. To keep one of them enabled, pass the matching --skip- argument to init.sh instead of turning the mechanism back on afterwards.
    • Keeping a mechanism enabled is a non-default configuration. The platform does not manage the traditional operating system, so an enabled security mechanism can cause unexpected failures on the node or in the cluster. You are responsible for operating and troubleshooting the nodes in this configuration.
    • When the host firewall is enabled, you configure and maintain the port rules on every node. For the ports that the platform requires, see Communication Matrix.
  • ✅ Disable swap .

• Users and Permissions

  • ✅ The node's SSH user has root privileges and can use sudo without the password.
  • ✅ The UseDNS parameter in /etc/ssh/sshd_config must be set to no.
  • ✅ Set the UsePAM parameter in /etc/ssh/sshd_config to no before adding the node, then restart sshd. PAM session policies (such as a forced password change, pam_access, faillock, or pam_limits) can otherwise block the SSH-based node join. After the node reaches the Ready state, you can restore UsePAM yes. On SELinux-enforcing systems that use password authentication, UsePAM no can itself break SSH login; use key-based authentication in that case.
  • ✅ systemctl show --property=DefaultTasksMax must return infinity; a low limit can make busy containers fail to create threads. If it is not infinity, set DefaultTasksMax=infinity in /etc/systemd/system.conf and run systemctl daemon-reexec.

• Node Network

  • ✅ hostname must comply with the following rules:
    • No more than 36 characters.
    • Starts and ends with a letter or number.
    • Contains only lowercase letters, numbers, -, and ., but cannot contain .-, .., or -..
  • ✅ localhost in /etc/hosts must resolve to 127.0.0.1.
  • ✅ The /etc/resolv.conf file must exist and contain nameserver configurations, but must not contain addresses starting with 172 (disable systemd-resolved).
  • ⚠️ The /etc/resolv.conf file should not configure search domains (if you must configure them, see Configure Search Domain).
  • ✅ The machine's IP address cannot be a loopback, multicast, link-local, all-0, or broadcast address.
  • ✅ Executing ip route must return a default route or a route pointing to 0.0.0.0.
  • ✅ The nodes must not occupy the following ports. This check confirms that the ports are free before the installation starts; it is not the list of ports that a host firewall must allow. For that list, see Communication Matrix.
    • Control plane nodes: 2379, 2380, 6443, 10249 ~ 10256
    • Node where the installer is located: 8080, 12080, 12443, 16443, 2379, 2380, 6443, 10249 ~ 10256
    • Worker nodes: 10249 ~ 10256
  • ✅ If the cluster uses Kube-OVN or Calico, ensure that the following ports are not occupied:
    • Kube-OVN: 6641, 6642
    • Calico: 179
  • ⚠️ Ensure that the IP addresses in the network segment 172.17.x.x ~ 172.18.x.x required by nerdctl are not occupied. If the IPs in this network segment are occupied and cannot be changed, please contact technical support.

• Software and Directory Requirements:

  • ✅ Must have the following installed: ip, ss, tar, swapoff, modprobe, sysctl, md5sum, and scp or sftp.
  • ⚠️ If you plan to use local storage TopoLVM or Rook, you need to install lvm2.
  • ✅ The /etc/systemd/system/kubelet.service file is not allowed to exist.
  • ✅ /tmp mount parameters must not contain noexec.
  • ✅ Remove packages that conflict with global cluster components (see Remove Conflicting Packages).
  • ✅ The following files must be deleted if they exist:
    • /var/lib/docker
    • /var/lib/nerdctl
    • /opt/nerdctl/
    • /var/lib/containerd
    • /var/log/pods
    • /var/lib/kubelet/pki

• Cross-Node Checks

  • ✅ Network communication between nodes in the global cluster must not be restricted:
    • In the default configuration, where no host firewall is enabled, there must be no network firewall restrictions between the nodes.
    • When a host firewall is enabled on the nodes, every port in Communication Matrix must be allowed between the nodes, and Pod and Service traffic inside the cluster must not be filtered by port.
  • ✅ The hostname of each node in the cluster must be unique.
  • ✅ The time zones of all nodes must be unified, and the time synchronization error must be ≤ 10 seconds.

#Appendix

#Hardened Node Security Configuration

Use this section only when a mandatory security policy, such as a STIG baseline, requires SELinux, AppArmor, or a host firewall to stay enabled on cluster nodes. Keeping all three disabled is the default and recommended node configuration, and it is what init.sh produces when you run it without arguments. Enabling any of them is a non-default configuration that you own, so validate it in a non-production environment before you apply it to production nodes.

#Keep the Mechanism Enabled Instead of Turning It Back On

init.sh can skip individual steps. Pass the matching --skip- argument so that the script never disables the mechanism your policy requires:

Combine the arguments you need. For example, to keep SELinux and firewalld enabled on a RHEL node:

bash init.sh --skip-disable_selinux --skip-disable_firewalld

This is the recommended path. The node keeps the state it was installed with, so no file system relabel and no extra reboot are required, and init.sh still performs every other node preparation step.

SELinux must end up in enforcing mode. permissive logs denials without blocking them, so it does not satisfy a policy that requires SELinux enforcement. It is still the safe intermediate step when you recover a node on which SELinux was already disabled, as described below.

#Order of Operations

Configure the node before the platform installation, not after. A policy or firewall problem then surfaces while the installer runs, instead of after the cluster carries workloads.

1. Run init.sh with the --skip- arguments for the mechanisms you want to keep enabled.
2. Configure the host firewall rules if a host firewall is enabled.
3. Verify the resulting state on the node, for example with getenforce, aa-status, and the status command of your firewall implementation.
4. Install the platform.

#If a Node Already Ran the Full init.sh

Use this path only when the mechanisms were already disabled and you have to turn them back on. It is more work than skipping the steps in the first place.

SELinux. A full init.sh run sets SELINUX=disabled and reboots the node. While SELinux is disabled the kernel does not maintain security labels, so files written to disk in that state carry no label. Editing /etc/selinux/config alone is not enough, and booting straight into enforcing on a file system that has only just been relabelled can leave the node unreachable: if the relabel is interrupted, or an early-boot file such as /etc/shadow or the sshd host keys ends up mislabelled, the node fails to boot or refuses SSH logins. Relabel under permissive first, and switch to enforcing only after the node comes back cleanly:

1. Set SELINUX=permissive in /etc/selinux/config.
2. Run touch /.autorelabel to request a full file system relabel on the next boot.
3. Reboot the node and wait for the relabel to finish. The node reboots again once the relabel completes, and comes up in permissive mode.
4. Review the denials logged during and after the relabel, for example with ausearch -m avc -ts boot, and resolve them before you continue.
5. Set SELINUX=enforcing in /etc/selinux/config and reboot the node.
6. Run getenforce and confirm that it returns Enforcing.

AppArmor. A full init.sh run stops and disables apparmor.service, and it can also add apparmor=0 to the kernel command line. Starting the service alone is therefore not always sufficient:

1. Check the kernel command line for apparmor=0. If it is present, remove it from the boot loader configuration and reboot the node.
2. Enable and start the service, for example with systemctl enable --now apparmor.
3. Run aa-status and confirm that the distribution default profiles are loaded and in enforce mode.

#Host Firewall

The platform does not require a specific host firewall implementation. RHEL and Kylin Linux Advanced Server nodes typically use firewalld, and Ubuntu nodes typically use ufw. Whichever implementation you use, allow the same set of ports. For the port list, see Communication Matrix.

The node that runs the installer also uses 8080, 12080, 12443, and 16443. Allow these ports on that node before you start the installation.

#Policy Sets

Distribution default policy sets are the supported path: the operating system vendor's targeted policy for SELinux, and the distribution default profiles for AppArmor.

Policy sets hardened beyond the distribution defaults, such as the STIG profile provided by Ubuntu Security Guide, and custom SELinux modules or booleans, are outside the validated range. Assess and validate them yourself before you apply them to cluster nodes.

#Remove Conflicting Packages

Before installation, applications may already be running in the docker/nerdctl/containerd environment on the nodes, or software conflicting with the global cluster may have been installed. Therefore, it is necessary to check and uninstall conflicting packages.

The following commands can be used for reference.

for x in \
    docker docker-client docker-common docker-latest \
    podman-docker podman \
    runc \
    containernetworking-plugins \
    apptainer \
    kubernetes kubernetes-master kubernetes-node kubernetes-client \
    ; do
    rpm -qa | grep -F "$x"
done

for x in \
    docker docker-client docker-common docker-latest \
    podman-docker podman \
    runc \
    containernetworking-plugins \
    apptainer \
    kubernetes kubernetes-master kubernetes-node kubernetes-client \
    ; do
    yum remove "$x"
done

for x in \
    docker.io \
    podman-docker \
    containerd \
    rootlesskit \
    rkt \
    containernetworking-plugins \
    kubernetes \
    ; do
    dpkg-query -l | grep -F "$x"
done

for x in \
    kubernetes-worker \
    kubectl kube-proxy kube-scheduler kube-controller-manager kube-apiserver \
    k8s microk8s \
    kubeadm kubelet \
    ; do
    snap list | grep -F "$x"
done

for x in \
    docker.io \
    podman-docker \
    containerd \
    rootlesskit \
    rkt \
    containernetworking-plugins \
    kubernetes \
    ; do
    apt-get purge "$x"
done

for x in \
    kubernetes-worker \
    kubectl kube-proxy kube-scheduler kube-controller-manager kube-apiserver \
    k8s microk8s \
    kubeadm kubelet \
    ; do
    snap remove --purge "$x"
done

for x in \
    docker docker-client docker-common \
    docker-engine docker-proxy docker-runc \
    podman-docker podman \
    containernetworking-plugins \
    apptainer \
    containerd \
    kubernetes kubernetes-master kubernetes-node kubernetes-client kubernetes-kubeadm \
    ; do
    rpm -qa | grep -F "$x"
done

for x in \
    docker docker-client docker-common \
    docker-engine docker-proxy docker-runc \
    podman-docker podman \
    containernetworking-plugins \
    apptainer \
    containerd \
    kubernetes kubernetes-master kubernetes-node kubernetes-client kubernetes-kubeadm \
    ; do
    yum remove "$x"
done

#Configure Search Domain

In Linux OS, the /etc/resolv.conf file is used to configure DNS client domain name resolution settings. The search line specifies the domain search path for DNS queries.

Configuration Requirements

• Number of Domains: The number of domains in the search line should be less than domainCountLimit - 3 (default domainCountLimit is 32).
• Length of Single Domain: Each domain name must not exceed 253 characters.
• Total Character Length: The total character count of all domain names and spaces must not exceed MaxDNSSearchListChar (default is 2048).

Example

search domain1.com domain2.com domain3.com

• The total number of domains is 3.
• The length of a single domain, such as domain1.com, is 11.
• The total character length is 35, i.e., 11 + 11 + 11 + 2 (two spaces).