[label bot] Continuously train and deploy label bot model - using GitOps

[jlewi]

We'd like to periodically (weekly) retrain the model and deploy the latest model to production.

This would allow us to benefit from

• Additional data to train from
• Incorporate new labels that might have been added

We'd like to use GitOps. We should automatically create a PR to update the model and once that model is approved and merged the new model should get deployed automatically.

There are three pieces

1. Automating training
   • Currently this means automating running the notebook that trains an AutoML model
2. Automatically creating a PR to update the model config to use the latest AutoML model
3. Use GitOps tools to automatically sync the latest model configs down to the cluster

The last step was taken care of by #152

For step 2 here's what I'm thinking create a tekton task that does the following

• Run a custom binary to get the latest deployed model from AutoML and emit it to a YAML file
• Use yq and kpt to update the label bot config
• Use hub CLI to create a PR

One thing I'm not sure about yet is how to perform the above in a reconcile loop; i.e. only trigger the above if the config isn't already pointing at the latest model

[jlewi]

I created a GitHub app to be used to create PRs
https://github.com/organizations/kubeflow/settings/apps/label-bot-cd

Created a private key
Finger print a0:3c:a9:87:e9:46:df:55:71:92:87:a1:bc:71:d6:6b:e0:7a:2f:89

Private key is stored in secret manager
https://cloud.console.google.com/security/secret-manager/secret/label-bot-cd-github-app-pem?project=issue-label-bot-dev

Installed it on the kubeflow/code-intelligence repo

• app-id: 71466

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/feature	0.95

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.