Developer Guide
This document is a guide for developers detailing the OPCT solution, design choices and the implementation references.
Table of Contents:
- Create Releases
- Development Notes
- Sonobuoy Plugins
- Diagrams
- CLI Result filters
- Running Customized Plugins
- Project Documentation
Create Release
Releasing a new version of the tool is done automatically through GitHub Action workflow which is run when tags are created. Tags should follow the Semantic Versioning (SemVer) standard. Example: v0.1.0, v0.1.0-alpha1 (...).
Tags should only be created from the main
branch which only accepts pull-requests that pass through this CI GitHub Action.
The container image for CLI is automatically published to the container registry quay.io/ocp-cert/opct
by Github Action job Build Container and Release to Quay
every new release.
Note that any version in v0. will be considered part of the preview release* of the tool.
Release process checklist:
- Make sure the base image has Security issues on Quay Security Scan. Steps to check:
- Build a "dev tag" for Plugin image:
cd openshift-tests-provider-cert && make build-dev
- Check the Security Scan results on https://quay.io/repository/ocp-cert/openshift-tests-provider-cert?tab=tags
- if there is security scan needed to fix on the base image, you must build the base image and tools image with the following steps:
-
- bump the version with fixes on the base image. Example: provider-certification-plugins#41
-
- build the tools image:
./hack/build-image.sh build-tools
- build the tools image:
-
- push the tools image:
podman push quay.io/ocp-cert/tools:v0.0.0-<the version>
- push the tools image:
-
- check security scan results
-
- if success, promote to latest:
podman push quay.io/ocp-cert/tools:latest
- if success, promote to latest:
-
- rebuild the plugins image with new base
- Build a "dev tag" for CLI image:
make linux-amd64-container IMG=quay.io/my-user/opct
podman push quay.io/my-user/opct:latest
- Check the Security Scan results on https://quay.io/repository/my-user/opct?tab=tags
- Build a "dev tag" for Plugin image:
- Run one-shot/preflight before promoting the tag:
- install a cluster using the latest OCP GA version for validation (platform agnostic installation)
- run the conformance validation (regular or upgrade)
- collect the results
- review if the execution finished correctly
- plugins finished
- artifacts collected
- results are under regular CI executions
- Create a tag on Plugins repository based on the
main
branch (or the commit for the release);# Example git tag v0.4.0 -m "Release for OPCT v0.4 related to features on OPCT-XXX" git push --tags upstream
- Open a PR updating the
PluginsImage
value on the CLI repository, merge it; - Create a tag on CLI/Tool repository based on the
main
branch (or the commit for the release)# Example git tag v0.4.0 -m "Release for OPCT v0.4 related to features on OPCT-XXX" git push --tags upstream
Manual tests
- Create an OCP cluster
- Prepare the cluster to run OPCT: set tests label for dedicated node, taint, create registry, create MachineConfigPool for upgrade, wait to be ready, etc
- It's possible to use the Ansible Playbook to do everything in Day-2 by running (WIP on #38):
ansible-playbook hack/opct-runner/opct-run-tool-preflight.yaml -e cluster_name=opct-v040
- It's possible to use the Ansible Playbook to do everything in Day-2 by running (WIP on #38):
- Run the tool
./opct-linux-amd64 run -w --plugins-image=openshift-tests-provider-cert:v0.4.0-beta2
- Collect the results and compare with old releases. Baseline CI artifacts is available here.
Development Notes
This tool builds heavily on Sonobuoy therefore at least some high level knowledge of Sonobuoy is needed to really understand this tool. A good place to start with Sonobuoy is its documentation.
OPCT extends Sonobuoy in two places:
- Command line interface (CLI)
- Plugins
Command Line Interface
Sonobuoy provides its own CLI but it has a considerable number of flags and options which can be overwhelming. This isn't an issue with Sonobuoy, it's just the result of being a very flexible tool. However, for simplicity sake, the OPCT extends Sonobuoy CLI with some strong opinions specific to the realm validating OpenShift on new infrastructure.
Integration with Sonobuoy CLI
The OPCT's CLI is written in Golang so that extending Sonobuoy is easily done. Sonobuoy has two specific areas on which we build on:
- Cobra commands (e.g. sonobuoy run)
- Sonobuoy Client (source code)
Ideally, the OPCT's commands will interact with the Sonobuoy Client API. There may be some situations where this isn't possible and you will need to call a Sonobuoy's Cobra Command directly. Keep in mind, executing a Cobra Command directly adds some odd interaction; this should be avoided since the ability to cleanly \ set Sonobuoy's flags may be unsafe in code like below. The code below won't fail at compile time if there's a change in Sonobuoy and there's also no type checking happening:
// Not Great
runCmd.Flags().Set("dns-namespace", "openshift-dns")
runCmd.Flags().Set("kubeconfig", r.config.Kubeconfig)
Instead, use the Sonobuoy Client includes with the project like this:
// Great
reader, ec, err := config.SonobuoyClient.RetrieveResults(&client.RetrieveConfig{
Namespace: "sonobuoy",
Path: config2.AggregatorResultsPath,
})
Sonobuoy image mirroring
The Sonobuoy images for Aggregator server and worker are mirrored to quay.io to prevent issues with docker hub network throttling.
The Sonobuoy image must follow the same of OPCT CLI (sonobuoy library defined in go.mod).
Running the mirror
The mirror steps must be executed every time the Sonobuoy package is update on CLI, the following steps describes how to start the mirror process:
- Update the version
SONOBUOY_VERSION
inhack/image-mirror-sonobuoy/mirror.sh
- Run mirror script
make image-mirror-sonobuoy
- Check the image in quay.io/opct/sonobuoy
Running the mirror targeting custom repository
SONOBUOY_VERSION=v0.56.11 MIRROR_REPO=quay.io/mrbraga/sonobuoy make image-mirror-sonobuoy
Adding new Architecture
If you are looking to test a new platform, you must add it when creating the manifest in hack/image-mirror-sonobuoy/mirror.sh
:
BUILD_PLATFORMS+=( ["windows-amd64"]="windows/amd64" )
Testing custom sonobuoy images in unsupported architectures
~/opct/bin/opct-devel run -w \
--sonobuoy-image quay.io/mrbraga/sonobuoy:v0.56.12-linux-arm64 \
--plugins-image openshift-tests-provider-cert:v0.5.0-alpha.3
Sonobuoy Plugins
OPCT is extended by Sonobuoy Plugins.
The Plugins source code is available on the project provider-certification-plugins.
Diagrams
The diagrams are available under the page Diagrams.
CLI Result filters
The CLI currently implements a few filters to help the reviewers (Partners, Support, Engineering teams) to find the root cause of the failures. The filters consumes the data sources below to improve the feedback, by plugin level, when using the command process
:
- A.
"Provider's Result"
: This is the original list of failures by the plugin available on the commandresults
- B.
"Suite List"
: This is the list of e2e tests available on the respective suite. For example: pluginopenshift-kubernetes-conformance
uses the suitekubernetes/conformance
- C.
"Baseline's Result"
: This is the list of e2e tests that failed in the baseline provider. That list is built from the same Certification Environment (OCP Agnostic Installation) in a known/supported platform (for example AWS and vSphere). Red Hat has many teams dedicated to reviewing and improving the thousands of e2e tests running in CI, that list is constantly reviewed for improvement to decrease the number of false negatives and help to look for the root cause. - D.
"Sippy"
: Sippy is the system used to extract insights from the CI jobs. It can provide individual e2e test statistics of failures across the entire CI ecosystem, providing one picture of the failures happening in the provider's environment. The filter will check for each failed e2e if has an occurrence of failures in the version used to be validated.
Currently, this is the order of filters used to show the failures on the process
command:
A intersection B
->Filter1
Filter1 exclusion C
->Filter2
Filter2 exclusion D
->Filter3
The reviewers should look at the list of failures in the following order:
Filter3
Filter2
Filter1
A
The diagram visualizing the filters is available on draw.io, stored on the shared Google Driver Storage, needing one valid Red Hat account to access it (we have plans to make it public soon): - https://app.diagrams.net/#G1NOhcF3jJtE1MjWCtbVgLEeD24oKr3IGa
Running Customized Plugins
In some situations, you may need to modify the plugins that are run by the OPCT. Running the OPCT with customized plugin manifests cannot be used for final validation of an OpenShift cluster! If you find issues or changes that are needed to complete, please open a GitHub issue or reach out to your Red Hat contact assisting with validation process.
- Export default plugins to local filesystem:
$ ./opct assets /tmp INFO[2022-06-16T15:35:29-06:00] Asset openshift-conformance-validated.yaml saved to /tmp/openshift-conformance-validated.yaml INFO[2022-06-16T15:35:29-06:00] Asset openshift-kube-conformance.yaml saved to /tmp/openshift-kube-conformance.yaml
- Make your edits to the exported YAML assets:
vi /tmp/openshift-kube-conformance.yaml
- Launch the tool with customized plugin:
./opct run --plugin /tmp/openshift-kube-conformance.yaml --plugin /tmp/openshift-conformance-validated.yaml
Project Documentation
The documentation is available in the directory docs/
. You can render it as HTML using mkdocs
locally - it's not yet published the HTML version.
To run locally you should be using python >= 3.8
, and install the mkdocs
running:
pip install hack/docs-requirements.txt
Then, under the root of the project, run:
mkdocs serve
Then you will be able to access the docs locally on the address: http://127.0.0.1:8000/