Hugging Face pattern matching and allowedOrganization support #2041
Conversation
It's invalid to put a wildcard in the HF org (`foo*bar/`) or omit the model name (`foo/`). Signed-off-by: Paul Boyd <paul@pboyd.io>
Extends the Hugging Face source to support wildcard patterns like: - org/* (all models from organization) - org/prefix* (models with specific prefix) This was already supported when previewing a source. Signed-off-by: Paul Boyd <paul@pboyd.io>
Signed-off-by: Paul Boyd <paul@pboyd.io>
|
/cc @adysenrothman |
adysenrothman
left a comment
adysenrothman
left a comment
There was a problem hiding this comment.
/lgtm
Tested locally and looks good! Thanks, Paul.
| **Valid patterns:** | ||
| - `"org/model"` - Exact model name | ||
| - `"org/prefix*"` - Models starting with prefix | ||
| - `"org/*"` - All models from organization | ||
| - `"*"` - All models (only when using `allowedOrganization`) | ||
|
|
||
| **Invalid patterns (will be rejected):** | ||
| - `"*"` - Global wildcard (without `allowedOrganization`) | ||
| - `"*/*"` - Global organization wildcard | ||
| - `"org*"` - Wildcard in organization name | ||
| - `"org/"` - Empty model name | ||
| - `"*prefix*"` - Multiple wildcards |
There was a problem hiding this comment.
Nit: Doesn't mention if it's valid to have wildcard at beginning or middle: *something or some*thing. Maybe it's implied well enough. Just noting in case we want to add now or in a follow-up
There was a problem hiding this comment.
Good call out. I pushed a commit that attempts to clarify this. includedModels only allows a prefix or an exact name, but excludedModels supports wildcards anywhere.
Signed-off-by: Paul Boyd <paul@pboyd.io>
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: pboyd The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
3 participants
Description
Support wildcards for including Hugging Face models and add an
allowedOrganizationproperty to limit access to a single org.Wildcards already worked when previewing a source, but this brings it into the normal flow.
How Has This Been Tested?
Locally on a kind cluster.
Merge criteria:
DCOcheck)