Post-Install Tasks

Congratulations you’ve successfully created an OpenShift cluster in a disconnected environment!

🪩 💃 It’s DISCO time! 🕺 🪩

Or, perhaps, your cluster will finish installing in the next ~30 minutes…​ 😸

Don’t worry! Your workshop environment includes a previously-built salsa cluster, which is also disconnected. You can complete this lab using either cluster.

disco diagram

Are we done yet?

Installing OpenShift in a disconnected environment concludes the "day 1" tasks. However, a disconnected cluster is more useful when you make a few simple configuration changes. These are commonly referred to as "day 2" tasks.

Here are a few suggested "day 2" tasks that will improve your experience.

1. Disable the default app / Operator CatalogSources

2. Create your own app / Operator CatalogSource

3. Install additional ImageContentSourcePolicies

4. Update OpenShift from a disconnected Update Server

You may not have enough time during the Summit 2024 Lab Session to complete all of these tasks.

Accessing your cluster

Because of time constraints, we recommend that you complete this lab using the pre-built salsa.lab cluster. If you complete quickly, and your disco.lab cluster has finished installing, you can repeat the steps again.

Retrieving the default kubeadmin password

The OpenShift install process creates a randomized password for the default cluster administrative user, kubeadmin. The password is printed to the screen when openshift-install finishes installing the cluster.

If you don’t want to wait for the cluster to finish installing, or your terminal output is no longer available, you can find the password by looking in the openshift-install directory.

Use the available tmux pane to find the kubeadmin password for your highside disco.lab OpenShift cluster.

echo "URL: https://console-openshift-console.apps.disco.lab"
echo "Username: kubeadmin"
echo "Password: $(more /mnt/high-side-data/auth/kubeadmin-password)"
URL: https://console-openshift-console.apps.disco.lab
Username: kubeadmin
Password: *****-*****-*****-*****

Log in to OpenShift using the 'oc' command-line tool

SSH to the appropriate system from the jump system, then use oc to log in to the OpenShift cluster.

The pre-built salsa.lab OpenShift cluster has simplified credentials.

  • A cluster-admin user is available as admin / admin.

  • A non-admin user is also available as user / user.

  • Its mirror-registry credentials are init / salsapass

Both of your tmux panes may be logged in to the highside system. You may need to type exit to go back to the jump system.

Logging in to either OpenShift cluster with the oc command-line tool will display an untrusted TLS certificate warning. You can safely answer y as in yes to continue logging in.
Using the pre-built salsa.lab cluster Using your own disco.lab cluster 
ssh salsa
oc login https://api.salsa.lab:6443 --username admin --password admin
 
ssh highside
oc login https://api.disco.lab:6443 --username kubeadmin
 
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

WARNING: Using insecure TLS client config. Setting this option is not supported!

Login successful.

You have access to 68 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
Welcome! See 'oc help' to get started.

Log in to the OpenShift Web Console

Please click on the Desktop button at the top of the right-hand terminal and use Firefox to log in to the OpenShift Web Console. Your workshop environment includes browser bookmarks to quickly connect with both of the OpenShift Web Consoles. Remember, the salsa cluster’s credentials are admin / admin.

As a convenience, the jump system has been configured to automatically trust both of the salsa.lab and disco.lab OpenShiftWeb Console certificates, but not the command-line / API certificates.
Screenshot of Desktop with DISCO - OpenShift bookmark highlighted

The Web Console is the very last part of OpenShift to come online. If you are trying to connect to your disco.lab cluster, and see an "Application is not available" message, it means the installation hasn’t completed yet.

The web-based VNC client (noVNC) is a bit klunky with copy and paste. You can copy and paste into the desktop using noVNC's tool ribbon (hidden by default.) You might need this if you’re trying to copy and paste the 23-character kubeadmin password.

Just click the Clipboard icon in the tool ribbon, paste your kubeadmin password, and then click the Clipboard icon again to save your password in the copy buffer & dismiss the dialog. Now you can go paste your password in the cluster login box.
Screenshot of Desktop with noVNC clipboard highlighted

Install Operators in a disconnected environment

Operators are like apps. OpenShift’s OperatorHub is like an app store. The OperatorHub is configured, by default, to show you apps / Operators from multiple sources. The sources include:

  • Operators that are provided by, and supported by, Red Hat

  • Operators that are provided by, and supported by, Certified Partners (Marketplace)

  • Operators that are provided by Certified Partners without support

  • Operators that are provided by the Community without support

Each source provides its own catalog of Operators. OpenShift will check each source for new catalog data (new Operators) every 10 minutes.

When oc-mirror downloads Operators, it automatically creates catalogs and sources. The catalogs are automatically uploaded into the mirror-registry. But the sources, which are YAML files tell OpenShift where to find the catalogs, need to be added to OpenShift manually as a "day 2" task.

Add custom CatalogSources

In a disconnected environment, you must disable the default CatalogSources.

The default CatalogSources can be disabled individually, or all at once. The command below will disable all of the default sources in a single command.

oc patch OperatorHub cluster --type merge -p '{"spec": {"disableAllDefaultSources": true}}'
operatorhub.config.openshift.io/cluster patched

The CatalogSources created by oc-mirror only contain the Operators that were mirrored (along with their dependencies). All the other Operators are removed from the catalog because they weren’t mirrored and can’t be installed.

After the default CatalogSources have been disabled, you can add the disconnected CatalogSources that oc-mirror created to your cluster.

The oc-mirror results files for the salsa.lab cluster were created in the lab-users home directory.

The oc-mirror results files for the disco.lab cluster were created in the /mnt/high-side-data directory.

Please adjust the following command accordingly.

 
oc create -f $HOME/oc-mirror-workspace/results-*/catalogSource-cs-redhat-operator-index.yaml
catalogsource.operators.coreos.com/cs-redhat-operator-index created

The new CatalogSources will appear in the Web Console / OperatorHub after ~60 seconds. You may notice that the salsa.lab cluster’s list of Operators includes the OpenShift Update Service and Logging Operators in addition to the Web Terminal Operator. The DevWorkSpace Operator is a dependency of the Web Terminal Operator.

The OpenShift Web Console will automatically hide the left-hand navigation menu when the browser window is not wide enough. You can reveal the navigation menu, and find the OperatorHub link, by clicking the "hamburger" menu in the top-left corner.
Screenshot of the salsa.lab clusters OpenShift Web Console showing OperatorHub with a custom CatalogSource

Install the Web Terminal Operator

Go ahead and install the Web Terminal Operator. Accept all of the default values. Your disconnected cluster will pull all of the images from your disconnected mirror-registry.

After the Web Terminal Operator has been installed, refresh your browser page to see the new terminal icon it provides at the top of the page. Clicking the terminal icon will create a terminal in your browser that is logged in to the cluster with your current credentials and permissions. The terminal includes all of the standard oc, kubectl, helm, and bash tools.

Screenshot of the Web Terminal Operator installed and running

Update your cluster

There are multiple ways to update your disconnected OpenShift clusters.

The easiest and fastest way to apply OpenShift updates is to use the oc adm upgrade command and reference the new version / release image. You will use this method below to update your cluster.

Installing and configuring the OpenShift Update Service Operator is discussed in the Optional Lab.

OpenShift versions are also known as releases. The list of updates, changes and dependencies each release / version provides is bundled into a release image and stored in your mirror-registry.

"Applying an OpenShift update" could also be described as "moving to a new release."

Before OpenShift applies any updates, it first checks the update / release’s signature to make sure it came from a trusted source. oc-mirror automatically downloads signatures for each OpenShift version / release that it downloads.

Another "day 2" task you must complete is uploading the release signatures from oc-mirror’s results files into your OpenShift cluster.

The oc-mirror results files for the salsa.lab cluster were created in the lab-users home directory.

The oc-mirror results files for the disco.lab cluster were created in the /mnt/high-side-data directory.

Please adjust the following command accordingly.

You can add the release signatures to your OpenShift cluster with this command.

oc apply -f $HOME/oc-mirror-workspace/results-*/release-signatures/
configmap/sha256-08b8725ce619ff1855cb0ec5f5c5baa879ef3c6ab9930db300761b97d2761144 created
configmap/sha256-e64464879cd1acdfa7112c1ac1d90039e1689189e0af197f34881c79decda933 created

OpenShift will complain if you tell it to update to a new release using the tag name. OpenShift prefers to update to new releases using the sha256 digest because the digest guarantees the release image’s contents haven’t been altered.

Please use the following commands to identify the sha256 digest and tell OpenShift to begin the update.

The username / password for the salsa mirror-registry is init / salsapass.

The username / password for the highside mirror-registry is init / discopass.

 
podman login -u init -p salsapass $(hostname):8443
podman login -u init -p discopass $(hostname):8443

DIGEST=$(oc image info -o json $HOSTNAME:8443/openshift/release-images:4.14.20-x86_64 | jq -r .contentDigest)
oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@$DIGEST --allow-explicit-upgrade
Login Succeeded!
Error: logging into "disco" because you're on "salsa"

warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
Requested update to release image quay.io/openshift-release-dev/ocp-release@sha256:e64464879cd1acdfa7112c1ac1d90039e1689189e0af197f34881c79decda933

Applying cluster updates to the salsa.lab and disco.lab clusters, which are Single Node OpenShift clusters, will result in the cluster becoming periodically unreachable.

Your cluster(s) become unreachable when the update process replaces the old console and apiserver pods with new versions.

OpenShift updates also update the nodes' operating system, Red Hat Enterprise Linux CoreOS. You get two updates for the price of one!

Your cluster(s) will become unreachable again when the Single Node of OpenShift reboots to apply kernel and other critical system updates.

You can follow along with the OpenShift update by using the following commands.

oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.18   True        True          50s     Working towards 4.14.19: 116 of 860 done (13% complete), waiting on etcd, kube-apiserver
watch oc get clusteroperators
NAME                      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication            4.14.19   True        False         False      3h5m
...
console                   4.14.19   True        False         False      3h16m
...
kube-apiserver            4.14.20   True        False         False      3h23m
kube-controller-manager   4.14.19   True        True          False      3h25m   NodeInstallerProgressing: 1 nodes are at revision 6; 0 nodes have achieved new revision 7
kube-scheduler            4.14.19   True        True          False      3h25m   NodeInstallerProgressing: 1 nodes are at revision 7; 0 nodes have achieved new revision 8...
...