OpenShift Update Service

This section is a work-in-progress.

Please be patient, and leave your feedback on how it can be improved.

Overview

The OpenShift Update Service Operator (OSUS) provides a fleet of disconnected OpenShift clusters with centralized updates.

Red Hat maintains an update "graph" that describes the supported, and blocked, paths when upgrading an older version of OpenShift to a newer one. For example, the graph allows clusters to upgrade directly from version 4.14.0 to 4.14.20. However, clusters that are running OpenShift version 4.13.0 are instructed, by the graph, to first update to 4.13.39 before applying the 4.14.20 update.

oc-mirror will download the graph data if the ImageSetConfiguration sets mirror.platform.graph: true

For example:

---
apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
storageConfig:
  local:
    path: /home/lab-user/
mirror:
  platform:
    graph: true   # <-- this is required for OpenShift Update Service
    channels:
      - name: stable-4.13
        minVersion: 4.13.0
        maxVersion: 4.13.40
      - name: stable-4.14
        minVersion: 4.14.0
        maxVersion: 4.14.22
  ...

The salsa.lab cluster has already downloaded the graph data and the OpenShift Update Service Operator (also known as cincinnati).

This lab will show how to install and configure the Update Service Operator, and configure the disco.lab cluster to pull updates from it. The abbreviated list of tasks include:

1. Install the OpenShift Update Service (OSUS) Operator

2. Configure the OSUS pods to trust the salsa mirror-registry

3. Configure the salsa.lab to trust OSUS

4. Configure the disco.lab cluster to use OSUS:

Trust the TLS certificate of the salsa.lab cluster (where OSUS runs)
Trust the TLS certificate of the salsa mirror-registry (where the updates are stored)
Add the salsa mirror-registry’s authentication data to disc.lab’s global pull secret.

Installing the OpenShift Update Service Operator

Please install the OpenShift Update Service on the salsa.lab cluster. Accept all of the default values.

Screenshot of the salsa.lab clusters OpenShift Web Console showing OperatorHub with a custom CatalogSource

After the Operator has finished installing, return to the command-line to apply the UpdateService result file that was created by oc-mirror.

The updateService.yaml file created by oc-mirror will create the UpdateService in the the default namespace.

The command below includes a -n openshift-update-service and creates the UpdateService in the correct namespace.

oc apply -n openshift-update-service -f $HOME/oc-mirror-workspace/results-*/updateService.yaml

updateservice.updateservice.operator.openshift.io/update-service-oc-mirror created

Configure OSUS to trust the salsa mirror-registry

After applying the updateService.yaml result file, OSUS will immediately try to pull the graph data and list of available updates / releases from the salsa mirror-registry. But the pods/containers running OSUS haven’t been configured to trust any additional TLS Certificate Authorities (CA).

You can see the error message by describing the UpdateService

oc describe UpdateService --all-namespaces

...
Status:
  Conditions:
    Message:               image.config.openshift.io.Spec.AdditionalTrustedCA.Name not set for image name cluster
    Reason:                NotConfigured
    Type:                  RegistryCACertFound
...

Configuring the OSUS pods running on the salsa.lab cluster to trust the salsa mirror-registry requires two steps. First, create a ConfigMap (in the openshift-config namespace) with the salsa mirror-registry’s TLS Certificate Authority in two different keys. The first key of the ConfigMap is the special name updateservice-registry. The second key of the ConfigMap is the DNS name of the salsa mirror-registry with the colon (two dots) replaced with two other dots (periods .) In other words, salsa.registry.com:8443 becomes salsa.registry.com..8443 because the colon : is a reserved character in YAML.

Replacing the colon with two periods is only required if your mirror-registry uses a port number, like 8443.

oc create configmap osus-ca-trust -n openshift-config \
  --from-file=updateservice-registry=$HOME/quay/quay-rootCA/rootCA.pem \
  --from-file=$(hostname -f)..8443=$HOME/quay/quay-rootCA/rootCA.pem

configmap/osus-ca-trust created

Then patch OpenShift’s Image Configuration so that the new ConfigMap is also used to trust TLS certificates.

oc patch image.config/cluster --type merge \
  --patch '{"spec": {"additionalTrustedCA": {"name": "osus-ca-trust"}}}'

image.config.openshift.io/cluster patched

Configure the salsa.lab to trust OSUS

The salsa.lab cluster should also be configured to trust the OSUS TLS certificate. This will allow the salsa.lab cluster to apply updates it serves to itself.

The OSUS TLS certificate is provided by OpenShift’s Ingress Controller / Router.

The salsa.lab cluster already has a cluster-wide CA trust bundle because the install-config.yaml included the mirror-registry’s Certificate Authority. We need to download the pre-existing cluster-wide CA trust (configmap/user-ca-bundle) to a file and add the Router Certificate Authority to the end of that file.

oc extract configmap/user-ca-bundle -n openshift-config

oc extract secret/router-certs-default -n openshift-ingress \
  --keys=tls.crt --to=- >> ca-bundle.crt

ca-bundle.crt
# tls.crt

Then replace the pre-existing cluster-wide CA trust (configmap/user-ca-bundle) with the updated list of Certificate Authorities.

oc set data configmap/user-ca-bundle -n openshift-config --from-file ca-bundle.crt

configmap/user-ca-bundle data updated

The salsa.lab cluster will now trust itself when applying updates.

Use the following commands to point salsa.lab to itself for updates. Begin by identifying the URL of the graph data being served by OSUS.

OSUS_URL=$(oc get -n openshift-update-service updateservice update-service-oc-mirror -o jsonpath='{.status.policyEngineURI}/api/upgrades_info/v1/graph{"\n"}')
echo $OSUS_URL

https://update-service-oc-mirror-route-openshift-update-service.apps.salsa.lab/api/upgrades_info/v1/graph

Then patch the graph data URL into the salsa.lab cluster.

oc patch clusterversion/version --type merge -p "{\"spec\":{\"upstream\":\"$OSUS_URL\"}}"

clusterversion.config.openshift.io/version patched

Finally, you can use the Web Console (under Administration and Cluster Settings) to choose an update. You can also use the command-line oc adm upgrade to show the next suggested upgrade.

Configure the disco.lab cluster to use OSUS

Configuring the disco.lab cluster to use OSUS from the salsa.lab cluster adds one extra step, updating the global pull secret with credentials for the salsa mirror-registry.

The salsa mirror-registry credentials are init / salsapass.

Log in to the highside system, discover the salsa mirror-registry’s DNS name, trust its TLS certificates,and add its credentials to the disco.lab pull secret with podman.

SALSA_REG=$(openssl s_client -connect salsa:8443</dev/null 2>/dev/null | awk '/^issuer/ {print $NF}')
echo $SALSA_REG

ip-10-0-6-85.us-west-2.compute.internal

Download the salsa mirror-registry and OSUS TLS certificates and trust them.

Use SSH to copy the TLS certificate bundle file that was created on the salsa system.

Look in the Table of Contents, or the Workshop Overview to find your unique password.

scp salsa:~/ca-bundle.crt .

ca-bundle.crt                         100%  3823    4.3MB/s   00:00

Configure the highside system to trust salsa’s mirror-registry.

sudo cp -v ca-bundle.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

'ca-bundle.crt' -> '/etc/pki/ca-trust/source/anchors/ca-bundle.crt'

Combine the disco.lab’s cluster-wide CA trust with the salsa certificates.

oc extract configmap/user-ca-bundle -n openshift-config --to=- >> ca-bundle.crt

# ca-bundle.crt

Replace disco.lab’s cluster-wide CA trust with the combined CA bundle

oc set data configmap/user-ca-bundle -n openshift-config --from-file ca-bundle.crt

configmap/user-ca-bundle data updated

Add the salsa mirror-registry credentials to the highside system’s local pull secret.

podman login --username init --password salsapass $SALSA_REG:8443
cat $XDG_RUNTIME_DIR/containers/auth.json

Login Succeeded!
{
  "auths": {
    "ip-10-0-54-198.us-west-2.compute.internal:8443": {
      "auth": "aW5pdDpkaXNjb3Bhc3M="
    },
    "ip-10-0-6-85.us-west-2.compute.internal:8443": {
      "auth": "aW5pdDpzYWxzYXBhc3M="
    }
  }
}

Replace the disco.lab cluster’s global pull secret

oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=$XDG_RUNTIME_DIR/containers/auth.json

secret/pull-secret data updated

Finally, patch the disco.lab cluster to look for updates from the OSUS on salsa.lab.

oc patch clusterversion/version --type merge \
  --patch '{"spec":{"upstream":"https://update-service-oc-mirror-route-openshift-update-service.apps.salsa.lab/api/upgrades_info/v1/graph"}}'

clusterversion.config.openshift.io/version patched

Confirm that disco.lab can update from OSUS running on salsa.lab cluster.

You can also use the Web Console (under Administration and Cluster Settings) to choose an update.

oc adm upgrade

Cluster version is 4.14.19

Upstream: https://update-service-oc-mirror-route-openshift-update-service.apps.salsa.lab/api/upgrades_info/v1/graph
Channel: stable-4.14 (available channels: candidate-4.14, candidate-4.15, eus-4.14, fast-4.14, fast-4.15, stable-4.14, stable-4.15)

Recommended updates:

  VERSION     IMAGE
  4.14.20     ip-10-0-6-85.us-west-2.compute.internal:8443/openshift/release-images@sha256:e64464879cd1acdfa7112c1ac1d90039e1689189e0af197f34881c79decda933

Your update / release will not be accepted if you haven’t uploaded the release signatures from the oc-mirror results files. See Lab 5 for details on how to trust the release signatures.

OSUS Appendix

ConfigMap YAML

oc get configmap/osus-ca-trust -n openshift-config

[lab-user@salsa ~]$ oc get -n openshift-config cm/osus-ca-trust -o yaml
apiVersion: v1
data:
  ip-10-0-6-85.us-west-2.compute.internal..8443: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  updateservice-registry: |
    -----BEGIN CERTIFICATE-----
    ....
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  name: osus-ca-trust
  namespace: openshift-config