Ability to use projectcalico.org/v3 custom resource definitions

[caseydavenport]

Description

This is a very early stage WIP PoC for removing the Calico API server and instead using v3 APIs directly as CRDs. This is solely meant as a way to scope out the required work in advance of a design.

Why?

• Calico API server constant source of bugs and burden.
• Differences in CRD vs v3 API
• Importing k8s libraries
• Massive amount of code needed for new APIs
• hostNetwork bugs on EKS, AKS.
• Must be running before APIs are accessible, HA, etc.

Rough work:

• Generate new v3 CRDs instead of crd.pc.org/v1
• Switch libcalico-go to read those CRDs instead of v1
• Assumptions reworked: e.g., tier prefixes
• Skip / disable various translations
• Add extensive kubebuilder validation
• Add tier-based RBAC using webhook
• Add IP Pool deletion controller for dependent resources
• Add short names to CRDs
• Add new v3 CRDs for ipamhandles / ipamblocks
• IP pool controller productized

Related issues/PRs

Tracking other issues I hit along the way!

• kubectl fails to apply strategic merge patch to custom resource kubernetes/kubernetes#97423

Todos

• Tests
• Documentation
• Release note

Release Note

Tech-preview support for using the projectcalico.org/v3 API directly as CRDs, without the need for an extension API server.

Reminder for the reviewer

Make sure that this PR has the correct labels and milestone set.

Every PR needs one docs-* label.

• docs-pr-required: This change requires a change to the documentation that has not been completed yet.
• docs-completed: This change has all necessary documentation completed.
• docs-not-required: This change has no user-facing impact and requires no docs.

Every PR needs one release-note-* label.

• release-note-required: This PR has user-facing changes. Most PRs should have this label.
• release-note-not-required: This PR has no user-facing changes.

Other optional labels:

• cherry-pick-candidate: This PR should be cherry-picked to an earlier release. For bug fixes only.
• needs-operator-pr: This PR is related to install and requires a corresponding change to the operator.

[james-callahan]

@caseydavenport is this something you expect to be merged/will it graduate from a PoC?

[caseydavenport]

@james-callahan yes, I am actively working on this now. I know the PR looks out of date but that's because there are a few other intermediate PRs I need to work on first that are prerequisites for this before I can pick this one up again.

I am hopeful that an alpha / tech-preview version of this can land in Calico v3.32.