From 62542737b21d40911a988e7c62c08379d41edc01 Mon Sep 17 00:00:00 2001 From: alexandre Date: Mon, 30 Mar 2020 12:13:01 -0400 Subject: [PATCH 1/2] added doc on testing operator bundles with OLM in 10 steps --- docs/testing-bundles.md | 246 ++++++++++++++++++++++++++++++++++++++++ index.md | 1 + 2 files changed, 247 insertions(+) create mode 100644 docs/testing-bundles.md diff --git a/docs/testing-bundles.md b/docs/testing-bundles.md new file mode 100644 index 0000000..28a49d8 --- /dev/null +++ b/docs/testing-bundles.md @@ -0,0 +1,246 @@ +## How do I Test My Operator Bundle? + +#### Operator Bundle in 10 Steps + +| Step | Description | +| --------- | ----------- | +| 1 | Make the changes you want to your CSV | +| 2 | Lint your bundle with operator-courier verify command | +| 3 | Test the bundle with `operator-sdk scorecard` command | +| 4 | If everything goes well, build operator bundle with operator-registry image + bundle | +| 5 | Push to a repository with a testing tag or name | +| 6 | Create a catalog source pointing to that bundle image | +| 7 | Create an operator group on the desired namespace | +| 8 | Create the subscription to your operator | +| 9 | Check InstallPlan, CSV and deployment | +| 10 | Finally, give a try on the operator via the UI | + +[Here](https://github.com/operator-framework/community-operators/blob/master/docs/using-scripts.md) you can find a few tips on how to build a makefile to automated your specific test. + + +#### Check the details below: +1. Make the changes you want to your CSV + +If you don't know what a CSV is begin [here](https://github.com/operator-framework/olm-book/blob/master/docs/packaging-an-operator.md#writing-your-operator-manifests). + +Boost your operator presentation putting good metadata content in your bundle. Check those 3 references: + +Documentation: + +- [CSV Reference](https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/building-your-csv.md) + +- [The OpenShift Descriptors](https://github.com/openshift/console/tree/master/frontend/packages/operator-lifecycle-manager/src/components/descriptors) +- [OpenShift Operator Hub UI on Steroids - x-Descriptors](https://github.com/openshift/console/blob/master/frontend/packages/operator-lifecycle-manager/src/components/descriptors/reference/reference.md) + +2. Lint your bundle with operator-courier verify command + +`operator-courier verify --ui_validate_io my-operator-bundle/` + +``` +If there is no output, the bundle passed operator-courier validation. +If there are errors, your bundle will not work. If there are warnings +we still encourage you to fix them before proceeding to the next step. +``` + +Documentation: + +[operator-courier](https://github.com/operator-framework/operator-courier) +[Testing your Operator](https://github.com/operator-framework/community-operators/blob/master/docs/testing-operators.md) + +* Please note that some commands have been updated and are not present on `testing your operator` documentation. Check the version release notes and the context help of your operator-sdk binary. But it's still a good source of information. + +3. Test the bundle with `operator-sdk scorecard` command: + + - Setup the `.osdk-scorecard.yaml` configuration file in the root of your project. [Check the docs](https://github.com/operator-framework/operator-sdk/blob/master/doc/test-framework/scorecard.md#config-file). + - Create your operator namespace if needed. + - Then, run the scorecard: `operator-sdk scorecard` + add the parameters you think may be useful. If your config file is in the root of your project it may be just that. Check the context help with `operator-sdk scorecard --help`. + - **Example**: Limit scorecard to `basic` test and directly provide kubeconfig: `operator-sdk scorecard --selector=suite=basic --kubeconfig ../auth/kubeconfig` + - Check the errors, if you get some, then debug them one by one until you have all with a `pass` status. Generally it's related to your CSV, but don't forget to include a version field to your CR Spec on your api and a Status field with valid values on your CR status field. + +An example of `.osdk-scorecard.yaml`: + +``` +scorecard: + output: json + bundle: "deploy/olm-catalog/nsm-operator/" + plugins: + - basic: + namespace: nsm + cr-manifest: + - "deploy/crds/nsm.networkservicemesh.io_v1alpha1_nsm_cr.yaml" + + - olm: + namespace: nsm + cr-manifest: + - "deploy/crds/nsm.networkservicemesh.io_v1alpha1_nsm_cr.yaml" + csv-path: "deploy/olm-catalog/nsm-operator/0.0.1/nsm-operator.v0.0.1.clusterserviceversion.yaml" +``` +Most common problems faced when running the scorecard tests and how to solve them: + +* Resources not listed under Custom Resource Definition. It will check if you have the types of resources owned by your primary resource. So let's say your operator deploys deamonsets, deployments, secrets etc it must have something like this: + +``` +... +spec: + customresourcedefinitions: + owned: + - description: A running Prometheus instance + displayName: Prometheus + kind: Prometheus + name: prometheuses.monitoring.coreos.com + resources: + - kind: StatefulSet + version: v1beta2 + - kind: Pod + version: v1 + - kind: ConfigMap + version: v1 + - kind: Service + version: v1 + specDescriptors: + ... +``` +* Status field not present. It will try to run an http get against your api to check if your status field have valid information. A common approach for your custom resource is to use a phase field with three or four states like `creating`, `running` or `terminating` for example. If you do change this make sure to at least `operator-sdk generate k8s` and `operator-sdk generate crds` in order to generate the necessary updates in your api base code in correct format. If you wish to go further you can validate against the openapi scheme using the [openapi-gen tool](https://github.com/OpenAPITools/openapi-generator). Check the sample code below. It will be under your pkg/api directory. + +``` +// NSMPhase is the type for the operator phases +type NSMPhase string + +// Operator phases +const ( + NSMPhaseInitial NSMPhase = "" + NSMPhasePending NSMPhase = "Pending" + NSMPhaseCreating NSMPhase = "Creating" + NSMPhaseRunning NSMPhase = "Running" + NSMPhaseTerminating NSMPhase = "Terminating" +) + +// NSMStatus defines the observed state of NSM +// +k8s:openapi-gen=true +type NSMStatus struct { + // Operator phases during deployment + Phase NSMPhase `json:"phase"` +} +``` +Then it needs to be updated by your controller at some point like that: + +``` + if nsm.Status.Phase == nsmv1alpha1.NSMPhaseInitial { + nsm.Status.Phase = nsmv1alpha1.NSMPhaseCreating + if updateErr := r.client.Status().Update(context.TODO(), nsm); updateErr != nil { + reqLogger.Info("Failed to update status", "Error", err.Error()) + } + } +``` + +* Scorecard Resources not garbage collected. The scorecard test should cleanup the Kubernetes objects it creates but prematurely terminating the test may cause some objects to remain. You may need to need to manually delete the following resources before running the test again: + +``` +oc delete secret scorecard-kubeconfig + +oc delete serviceaccount + +oc delete role/clusterrole + +oc delete rolebinding/clusterrolebinding + +oc delete deployment +``` + + +4. If everything goes well, build operator bundle with operator-registry image + bundle. +``` +podman build -t quay.io/my-container-registry-namespace/my-manifest-bundle: -f . +``` +Here is an example of a container configuration file for the registry-server. Please observe that the `COPY` command is copying your bundle manifests into a folder called manifests inside the image. So, don't forget to make them available in your path or change the path to suit your needs. + Observe that you can you use other container builders than podman as well. Consider buildah, it's quite interesting too! + +``` +FROM quay.io/operator-framework/upstream-registry-builder as builder + +COPY manifests manifests +RUN ./bin/initializer -o ./bundles.db + +FROM scratch +COPY --from=builder /build/bundles.db /bundles.db +COPY --from=builder /build/bin/registry-server /registry-server +COPY --from=builder /bin/grpc_health_probe /bin/grpc_health_probe +EXPOSE 50051 +ENTRYPOINT ["/registry-server"] +CMD ["--database", "bundles.db"] +``` +Documentation: + +[Operator Registry](https://github.com/operator-framework/operator-registry) + +5. Push to a repository with a testing tag or name + +`podman push -t quay.io/my-container-registry-namespace/my-manifest-bundle:` + +6. Create a catalog source pointing to that bundle image on the openshift-marketplace namespace: + +Here's an example: +``` +cat < Date: Tue, 7 Apr 2020 14:51:30 -0500 Subject: [PATCH 2/2] add info about v2schema2 requirement --- docs/testing-bundles.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/testing-bundles.md b/docs/testing-bundles.md index 28a49d8..6f2d1ec 100644 --- a/docs/testing-bundles.md +++ b/docs/testing-bundles.md @@ -176,6 +176,14 @@ Documentation: `podman push -t quay.io/my-container-registry-namespace/my-manifest-bundle:` +**Note:** Be sure to use the latest version of `podman` (or `docker`). This will ensure that your image has been pushed with the [v2schema2](https://docs.docker.com/registry/spec/manifest-v2-2/) format. You can verify this by running one of the following commands after successfully pushing the image: + +```skopeo inspect --raw --creds 'username':'password' docker:////: | grep schemaVersion``` + +```docker manifest inspect //: | grep schemaVersion``` + +The `schemaVersion` should be set to `2`. + 6. Create a catalog source pointing to that bundle image on the openshift-marketplace namespace: Here's an example: