Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 42 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,40 +2,62 @@
This repo is my setup/notes for running openshift/installer through ci-operator.
I use this to launch a cluster in AWS via openshift/installer, and to run CI against the cluster.
There are a few minor additions to the templates in this repo from those running in CI; I've
added a secret to transfer aws credentials, quay pull secret and the tectonic config.
added a secret to transfer aws credentials and quay pull secret.

First, get a token and login to the OpenShift CI cluster.
Next: Note the files required for the cluster-secrets-aws secret and create them/change paths
accordingly. The ssh key-pair is optional, and is required for OpenShift conformance tests to
pass.

Then:
1. oc new-project my-namespace
3. oc create secret -n your-namespace generic cluster-secrets-aws \
--from-file=secret/credentials \
--from-file=secret/pull-secret \
--from-file=secret/ssh-privatekey \
--from-file=secret/ssh-publickey \
-o yaml --dry-run | oc -n your-namespace apply -f -
4. ci-operator -template templates/cluster-launch-installer-e2e-new.yaml \
Next: Note the files required for the cluster-secrets-aws secret and populate their content.
The ssh key-pair is meant to be generated/only used for ci-testing. The private
key is used only when running the full conformance test suite, so that's optional.

Pick a short namespace name to avoid:

> the S3 bucket name
> "wking-next-gen-installer-ef260.origin-ci-int-aws.dev.rhcloud.com",
> generated from the cluster name and base domain, is too long; S3
> bucket names must be less than 63 characters; please choose a
> shorter cluster name or base domain

It looks like 22 characters is the max with the
`${NAMESPACE}-${JOB_NAME_HASH}` approach to cluster naming.

Then:

Create a new project in https://api.ci.openshift.org:
```bash
oc new-project your-namespace
```

Fill in contents of cluster-secrets-aws directory with contents of files:

(see files in `cluster-secrets-aws` directory for more info)
```bash
credentials (see noted format in this repo)
pull-secret (quay pull secret json config)
ssh-privatekey (only required for full conformance tests, not required for aws-e2e tests)
ssh-publickey (required by installer)
```

Now run the ci-operator command:
```bash
ci-operator -template templates/cluster-launch-installer-e2e-new.yaml \
-config /path/to/openshift/release/ci-operator/config/openshift/installer/master.yaml \
-git-ref=your-gh-username/installer@your-branch \
-secret-dir=/path/to/cluster-secrets-aws \
-namespace=your-namespace
```


You can then access your project in the CI web console, check the pods.
When running the smoke tests, you'll see the output of the smoke tests in
logs of pod/dev container/setup.

When running the full test suite, conformance test output will be found via
pod logs for pod/dev container/test.

You can grab the kubeconfig, copy to your local system, and access the cluster running in AWS.
From the CI web console in your project, go to pod/dev container/setup and in the terminal,
kubeconfig is at /tmp/shared/cluster/generated/auth/kubeconfig. Copy that to your local system, then
export KUBECONFIG=/path/to/copied/kubeconfig.
`export KUBECONFIG=/path/to/copied/kubeconfig`.

Clean up your AWS resources from a cluster!!!
1. git clone git@github.com:openshift/hive.git
2. cd hive; make hiveutil
3. bin/hiveutil aws-tag-deprovision --cluster-name your-cluster-name --loglevel debug tectonicClusterID=see-aws-console-for-tag
1. `git clone git@github.com:openshift/hive.git`
2. `cd hive; make hiveutil`
3. `bin/hiveutil aws-tag-deprovision --cluster-name your-cluster-name --loglevel debug tectonicClusterID=see-aws-console-for-tag`
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -7,27 +7,27 @@ parameters:
- name: JOB_NAME_HASH
required: true
- name: NAMESPACE
#required: true
value: "somalley-test"
# MODIFIED HERE, USE YOUR NAMESPACE
value: "your-ns"
- name: IMAGE_FORMAT
required: true
- name: IMAGE_INSTALLER
required: true
- name: IMAGE_TESTS
required: true
- name: CLUSTER_TYPE
required: true
# MODIFIED HERE, SPECIFY "aws"
#required: true
value: "aws"
- name: TEST_FOCUS
value: Suite:openshift/conformance/parallel
- name: TEST_SKIP
- name: TEST_FOCUS_SERIAL
value: Suite:openshift/conformance/serial
- name: TEST_SKIP_SERIAL
- name: TEST_PARALLELISM
value: "30"
# Ensures the release image is created and tested
- name: TEST_COMMAND
- name: RELEASE_IMAGE_LATEST
#- name: RELEASE_IMAGE_LATEST

objects:

Expand Down Expand Up @@ -61,22 +61,10 @@ objects:
emptyDir: {}
- name: shared-tmp
emptyDir: {}
- name: cluster-profile
secret:
secretName: ${JOB_NAME_SAFE}-cluster-profile
# MODIFIED HERE, added creds secret
- name: pull-secret-aws
secret:
secretName: cluster-secrets-aws
initContainers:
- name: cli
image: ${IMAGE_CLI}
volumeMounts:
- name: shared-tmp
mountPath: /tmp/shared
command:
- cp
- /usr/bin/oc
- /tmp/shared/oc

containers:

Expand All @@ -93,8 +81,6 @@ objects:
volumeMounts:
- name: shared-tmp
mountPath: /tmp/shared
- name: cluster-profile
mountPath: /tmp/cluster
- name: artifacts
mountPath: /tmp/artifacts
env:
Expand Down Expand Up @@ -184,10 +170,9 @@ objects:
volumeMounts:
- name: shared-tmp
mountPath: /tmp
- name: cluster-profile
mountPath: /etc/openshift-installer
- name: artifacts
mountPath: /tmp/artifacts
# MODIFIED HERE, added my creds secret
- name: pull-secret-aws
mountPath: /tmp/pull-secret-aws
env:
Expand All @@ -196,6 +181,7 @@ objects:
- name: TYPE
value: ${CLUSTER_TYPE}
- name: AWS_SHARED_CREDENTIALS_FILE
# MODIFIED HERE, added path to files in my secret
#value: /etc/openshift-installer/.awscred
value: /tmp/cluster/credentials
- name: KUBECONFIG
Expand All @@ -211,7 +197,7 @@ objects:
trap 'kill $(jobs -p); exit 0' TERM

mkdir /tmp/cluster
cp /etc/openshift-installer/* /tmp/cluster/
cp /tmp/pull-secret-aws/* /tmp/cluster/

mkdir /tmp/artifacts/installer
cd /tmp/cluster
Expand All @@ -221,8 +207,10 @@ objects:
OPENSHIFT_INSTALL_BASE_DOMAIN=origin-ci-int-aws.dev.rhcloud.com \
OPENSHIFT_INSTALL_EMAIL_ADDRESS=test@ci.openshift.io \
OPENSHIFT_INSTALL_PASSWORD=$( date +%s | sha256sum | base64 | head -c 32 ; echo ) \
OPENSHIFT_INSTALL_SSH_PUB_KEY="$( cat /etc/openshift-installer/ssh-publickey )" \
OPENSHIFT_INSTALL_PULL_SECRET="$( cat /etc/openshift-installer/pull-secret )" \
# MODIFIED HERE
# changed path to the pull-secret-aws files
OPENSHIFT_INSTALL_SSH_PUB_KEY="$( cat /tmp/cluster/ssh-publickey )" \
OPENSHIFT_INSTALL_PULL_SECRET="$( cat /tmp/cluster/pull-secret )" \
OPENSHIFT_INSTALL_PLATFORM=aws \
OPENSHIFT_INSTALL_AWS_REGION=us-east-1 \
openshift-install --dir /tmp/artifacts/installer --log-level debug cluster
Expand Down Expand Up @@ -272,10 +260,11 @@ objects:
volumeMounts:
- name: shared-tmp
mountPath: /tmp/shared
- name: cluster-profile
mountPath: /etc/openshift-installer
- name: artifacts
mountPath: /tmp/artifacts
# MODIFIED HERE, ADDED SECRET MOUNT
- name: pull-secret-aws
mountPath: /tmp/pull-secret-aws
env:
- name: INSTANCE_PREFIX
value: ${NAMESPACE}-${JOB_NAME_HASH}
Expand Down Expand Up @@ -336,7 +325,8 @@ objects:
wait

echo "Deprovisioning cluster ..."
export AWS_SHARED_CREDENTIALS_FILE=/tmp/shared/cluster/credentials
# MODIFIED HERE path
export AWS_SHARED_CREDENTIALS_FILE=/tmp/pull-secret-aws/credentials
openshift-install --dir /tmp/artifacts/installer --log-level debug destroy-cluster
}

Expand All @@ -349,4 +339,3 @@ objects:
fi
sleep 60 & wait
done

Loading