Upgrade the global cluster

Upgrade Path

This page covers the traditional operating system path for the global cluster. If your global cluster uses Alauda OS, the Kubernetes step lives in the Immutable Infrastructure documentation — see Upgrading the global Cluster on Immutable Infrastructure. The Core, Aligned, and Agnostic steps described on this page still apply to immutable-OS clusters; only the way Kubernetes is rolled out differs.

consists of a global cluster and one or more workload clusters. To move the platform to a new ACP Distribution Version, upgrade the global tier to the target Distribution Version first, and then upgrade workload clusters to that same Distribution Version.

Cluster upgrades use the CVO-based workflow. A typical global cluster upgrade includes artifact preparation, preflight checks, upgrade request, and status observation.

Before upgrading the global cluster, verify that every workload cluster is within the target release's Compatible Versions in the Kubernetes Support Matrix. This prerequisite is separate from the third-party cluster onboarding range.

This Compatible Versions prerequisite applies whether or not the environment uses global DR. Global DR changes the procedure used to upgrade the global tier, but it does not change the requirement that workload clusters must remain within the compatible Kubernetes version range before the global tier is upgraded to the target Distribution Version.

Global cluster upgrades follow the validated upgrade.sh-based procedure documented on this page. You can request the global-cluster upgrade from the Web Console, by updating ClusterVersionShadow.spec.desiredUpdate, or by using ACP CLI with --cluster=global. For the complete AC CLI workflow and output interpretation, see Upgrading Clusters. For full command and flag syntax, see AC CLI Administrator Command Reference.

If the environment uses global DR, follow Upgrade Global Clusters in a DR Environment. Otherwise, follow the standard workflow below.

Standard Workflow

A global cluster upgrade is staged across a timeline. Most of the work is completed before the maintenance window so the window itself stays short and predictable:

PhaseWhenWhat happensCluster impact
1. Sync artifactsAny time before the windowFor the platform built-in Registry, upgrade.sh --only-sync-image uploads Core artifacts and the eleven packages that you manually place in plugins/. For an external registry, manually upload the target Core payload before running upgrade.sh. In both cases, violet publishes the other Aligned and Agnostic packages needed for the window.None — images and catalog entries are published, but no running plugin is changed; no downtime.
2. Preflight1–2 weeks before the windowupgrade.sh --preflight validates upgrade readiness. Resolve every blocking item before the window opens.None — read-only validation.
3. UpgradeDuring the maintenance windowupgrade.sh --skip-sync-image deploys or updates the cluster version operator (CVO); the upgrade is then requested and observed.The cluster is upgraded.

Phase 1 and Phase 2 do not change cluster state — run them early so the maintenance window only contains Phase 3. A small or lab environment can instead run a single bash upgrade.sh that performs synchronization and CVO deployment together, but production windows usually keep them separate.

Sync upgrade artifacts

When: any time before the maintenance window. This step uploads artifacts to the registry and does not change cluster state.

First, record the configured registry address and determine whether the platform uses the built-in or an external registry:

kubectl get productbase base \
  -o jsonpath='{.spec.registry.address}{"\t"}{.spec.registry.external}{"\n"}'

The second value is false for the platform built-in Registry and true for an external registry. The publication command is selected after the target Aligned packages are copied into plugins/.

The Core Package does not contain the Aligned packages from ACP Upgrade to v4.5. Copy each of the eleven separately downloaded packages listed in Pre-Upgrade Preparation into the plugins/ directory of the extracted Core Package. Run the following command once for each package:

cp <path-to-upgrade-extension-package> ./plugins/

Confirm that the directory contains all eleven packages before synchronization:

ls -1 ./plugins/

Publish the staged payload by following the branch that matches the value of ProductBase.spec.registry.external.

For the platform built-in Registry (false), run upgrade.sh in sync-only mode from the extracted Core Package directory:

bash upgrade.sh --only-sync-image

--only-sync-image uploads Core images and the packages that you manually placed in plugins/, without deploying the cluster version operator or changing running applications. The CVO is deployed later, inside the maintenance window — see Deploy the cluster version operator.

For an external registry (true), do not use --only-sync-image as proof of publication. upgrade.sh automatically skips image synchronization. From the target Core Package's installer/ directory, run both res/upload.sh all and res/upload.sh necessary by following Prepare the Target-Version Payload. The modes may run in either order, but both must succeed.

The selected publication path makes the artifacts required by the CVO-based workflow available, including:

TypeContentPurpose
Product imagesproduct-imageUsed to resolve the target version and image in ProductManifest and CVO.
CVO imagecluster-version-operatorUsed to deploy or update the cluster version operator.
Plugin artifactsplugins/*.tgzUsed by the upgrade plan when plugin artifacts are required.

For the eleven Aligned applications from ACP Upgrade to v4.5, always stage the packages in plugins/. The built-in Registry path publishes them with upgrade.sh --only-sync-image; the external Registry path publishes them through res/upload.sh all. CVO upgrades an application only when it is already installed; publishing a package for an application that is not installed does not install it.

Use violet push for the other packages downloaded from the violet list inventory:

Package groupPublication pathRequirement
Other Aligned cluster pluginsviolet push to the global tierRequired for each installed plugin that is not part of ACP Upgrade to v4.5.
Other Aligned operatorsviolet push to the global tier and each workload cluster where the operator is installedRequired before upgrading each cluster that runs the operator.
Agnostic cluster plugins and operatorsviolet push to the clusters where you plan to upgrade themOptional for CVO; required before starting their separate Marketplace or operator upgrade.
# Publish another cluster plugin to the global tier
violet push <path-to-other-cluster-plugin-package> \
  --platform-address "https://<your-platform-domain>" \
  --platform-token "<platform_token>"

# Publish another operator to global and each workload cluster that runs it
violet push <path-to-other-operator-package> \
  --platform-address "https://<your-platform-domain>" \
  --platform-token "<platform_token>" \
  --clusters "global,<workload-cluster-1>,<workload-cluster-2>"

Pushing every operator package to every cluster from the global window is the recommended pattern. If the global window has already passed and you discover a workload cluster missing an operator package, you can still push to that workload cluster before the workload upgrade — see Upgrade workload clusters.

WARNING

Before requesting the upgrade, make the target package available for every Aligned plugin installed on the cluster. Use upgrade.sh for applications from ACP Upgrade to v4.5, and violet for the other Aligned packages. Synchronizing a package does not install an application that is not currently installed. For an installed Aligned cluster plugin, CVO requires the matching target ModulePluginConfig to be Ready; otherwise, CVO cannot build the upgrade plan, reports the error through ClusterVersionShadow, and retries the existing request. An installed Aligned operator requires a matching target InstallPlan from its catalog.

Registry behavior depends on how the environment is configured:

ScenarioBehavior
--registry is specifiedUse the provided registry directly.
--registry is not specifiedRead the registry address from ProductBase.spec.registry.address.
Built-in platform registryRebuild the access address by using the global VIP.
External registryAutomatically set SKIP_SYNC_IMAGE=true; the target payload must already have been uploaded through the canonical target-version payload procedure.
Image upload required but credentials omittedRead username and password from the cpaas-system/registry-admin Secret.

When the target registry is not the platform default, add registry parameters:

ParameterPurpose
--registrySpecify the target registry address.
--username / --passwordSpecify registry credentials.

To bypass artifact validation when it is known to be redundant, add --skip-check-artifacts.

WARNING

Do not open the maintenance window until image and plugin synchronization is complete.

Before running preflight, confirm that the registry configuration is still correct:

kubectl get productbase base \
  -o jsonpath='{.spec.registry.address}{"\t"}{.spec.registry.external}{"\n"}'

Use successful completion of the selected publication commands and the registry administration interface or API as the Phase 1 evidence. Confirm that the expected target repositories and tags exist, and compare the published Extension set with the installed inventory exported to apps.yaml. Do not use ProductBase.status.artifacts as a zero-output target-version gate: it represents the current catalog and can legitimately contain Absent entries for optional or uninstalled packages.

Run preflight checks

When: 1–2 weeks before the maintenance window, so there is time to resolve any blocking items before the window opens.

Run upgrade.sh in preflight mode:

bash upgrade.sh --preflight

Preflight is read-only — it validates upgrade readiness and does not change cluster state.

Preflight returns two parts:

OutputPurpose
SummaryShows the overall result, current version, desired version, and desired image.
ChecksShows the result of each individual validation item.

The default check set includes:

  • ResourcePatchUpgradeable
  • ClusterVersionUpgradeable
  • AdminAckRequired
  • VersionUpgradePath
  • KubernetesVersionSupported
  • DockerRuntimeUnsupported
  • ClusterRunning
  • ClusterModuleStable
  • ControlPlaneStaticPodsPresent
  • CustomEtcdBackupCronJobsAbsent
  • CRIUpgradePodsAbsent
  • ModuleInfoStable
  • PlatformLicense

Handle preflight blocks

When: before the maintenance window, as soon as preflight reports a block.

A clean first preflight run is not the norm on a customized environment. Treat resolving blocks as part of the preparation phase, not as an exception — that is why preflight is run one to two weeks ahead.

If any check does not pass, do not open the maintenance window. Resolve each block with Upgrade Troubleshooting, which covers:

Do not force an upgrade past a failed check. VersionUpgradePath, KubernetesVersionSupported, and AdminAckRequired gate the supported upgrade path itself. If a block cannot be resolved, contact technical support before continuing.

Deploy the cluster version operator

When: at the start of the maintenance window.

Run upgrade.sh in skip-sync mode. Synchronization is skipped because the artifacts were already uploaded in Sync upgrade artifacts:

bash upgrade.sh --skip-sync-image

This deploys or updates the cluster version operator (CVO) and completes the remaining preparation. The CVO drives the Core and Aligned plugin upgrade once the upgrade is requested in the next step.

After the command completes, inspect the target ProductManifest before requesting the upgrade:

kubectl get productmanifest v<target-version> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

kubectl get productmanifest v<target-version> -o json | jq -r '
  .status.artifacts[] as $artifact
  | $artifact.channels[]
  | select(.artifactStatus != "Ready")
  | [$artifact.name, .channel, .tag, .artifactStatus]
  | @tsv'

The per-artifact output is diagnostic and is not required to be empty. The target manifest can include optional, Agnostic, or uninstalled entries that are legitimately not Ready. Compare the output with the installed Core and Aligned inventory. For an installed Aligned ModulePlugin, inspect the matching target ModulePluginConfig when its channel is not Ready:

kubectl get modulepluginconfig <modulepluginconfig-name> -o yaml

Use the component name and channel tag from the target ProductManifest, or the name reported in the ClusterVersionShadow error, to identify the ModulePluginConfig. The component version in this name is not necessarily the ACP Distribution Version. Do not treat artifactStatus: Absent alone as proof that the registry manifest is missing; use the ModulePluginConfig condition and its .spec.image to distinguish a missing manifest from authentication, CA, network, or package-content failures.

Request the upgrade

After the cluster version operator is deployed, request the upgrade through one of the following entry points. The entry points are equivalent; pick the one that fits your operating model.

The platform offers two cluster management interfaces in the Web Console. Web Console is the built-in interface that ships with the platform and is the default path. Web Console (Fleet Essentials) is the newer cluster management interface, delivered as a separate cluster plugin that must be installed before it can be used.

Web Console
Web Console (Fleet Essentials)
ACP CLI
kubectl

This is the built-in cluster management interface. It is present on every platform and requires nothing to be installed beyond the platform itself.

Requesting the upgrade requires update permission on the target cluster's ClusterVersionShadow and ResourcePatch resources.

  1. In the left navigation bar, click Clusters > Clusters, then click the global cluster.

  2. Open the Functional Components tab.

    The tab lists each component with its Status and Version. A component that has a newer version available is marked Upgradeable. A component that cannot move to the new version carries a block reason instead, and the page shows which components are not compatible with the new version.

  3. Click Upgrade.

    The button is disabled when there is no upgradable version (No upgradable version), when an upgrade is already running (Upgrading), or when the cluster is not healthy (Cluster Abnormality). Hover over the button to read the reason.

  4. On the Personalized Configuration Confirmation step, review the listed configurations, then click Acknowledge.

    This step appears only when the cluster carries personalized configuration — a customized ModuleInfo, a customized ClusterModule, or a ResourcePatch. It lists each item and renders the selected item's overrides as read-only YAML, under the warning There are the following personalized configurations in your environment, please confirm with the technical support staff whether you need to delete or adjust, otherwise, your upgrade may fail!

    When the cluster carries no personalized configuration, this step does not appear and the page opens directly on the confirmation form described in the next step.

  5. On the Confirm Component Upgrade step, review Current Version and Target Version.

    The target version is the first entry of ClusterVersionShadow.status.availableUpdates. It is determined by the upgrade artifacts already prepared for the cluster and cannot be selected manually in this interface.

  6. Click Upgrade.

  7. In the Are you sure you want to upgrade the cluster version? dialog, set Maintenance mode, then click Upgrade to submit the request.

Maintenance mode is switched on by default in this dialog

The confirmation dialog opens with Maintenance mode already enabled. While maintenance mode is on, users who are not platform administrators are restricted from accessing platform features, and the text in Platform maintenance announcements is shown to them. The announcement is required while the switch is on and is limited to 200 characters. Turn the switch off before submitting if the change window does not call for restricting access. The switch writes the platform-wide maintenance mode setting, so review that setting again after the change window closes.

After the request is submitted, the page returns to the Functional Components tab and shows The upgrade has been initiated. It may take 5~10 minute for the status to update. Click the magnifier icon in a component's Status column to follow that component's upgrade progress.

Observe execution

Use the following command to inspect the overall status:

kubectl get cvsh -n cpaas-system

Important status fields:

FieldPurpose
status.conditionsOverall status entry point.
status.preflight.observedAtTime of the latest preflight run.
status.preflight.checksDetailed result of each preflight item.
status.currentCurrent applied version and image.
status.desiredTarget version and image being reconciled.
status.historyUpgrade history, newest first.
status.stagesUpgrade stages and per-stage execution state.

Focus on these conditions first:

ConditionInterpretation
PreflightReadyTrue means preflight passed.
ReadyTrue means the cluster has reached the desired version.
ReconcilingTrue means the upgrade is still running.
StalledTrue means the upgrade is blocked and requires intervention.

If Stalled=True, follow Upgrade Troubleshooting. To observe an individual plugin or operator module, read its ModuleInfo:

# Current version, target, next available version, and phase per installed module on this cluster
kubectl get moduleinfo -l cpaas.io/cluster-name=global \
  -o custom-columns='MODULE:.metadata.labels.cpaas\.io/module-name,CURRENT:.status.version,TARGET:.spec.version,NEW:.status.availableVersions[0].version,PHASE:.status.phase'

# Conditions and any block reasons for one module
kubectl get moduleinfo <name> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

A module has reached its target when status.phase is Running and status.version equals the target version.

Upgrade Agnostic plugins from Marketplace

CVO drives Core and Aligned plugins. Agnostic plugins are outside CVO's scope and must be upgraded individually after the cluster has reached the target Distribution Version. Whether each Agnostic plugin needs to be upgraded depends on its own Kubernetes compatibility — see the plugin's release notes for compatibility with the target Kubernetes version.

For each in-use Agnostic plugin on the global cluster:

  1. In the Web Console, switch to Administrator view.
  2. Navigate to Marketplace > Cluster Plugins for Agnostic cluster plugins, or to the operator workflow for Agnostic operators.
  3. Select the target plugin or operator and trigger the upgrade. The Marketplace upgrade flow reads the package previously pushed with violet.

If you skipped pushing an Agnostic plugin during pre-upgrade and the Marketplace does not offer the target version, complete the violet push step first and then retry the Marketplace upgrade.

Validate the Upgrade

After the cluster reaches the desired version and the in-use Agnostic plugins are handled, complete Upgrade Validation.