Conversation
This reverts commit 6ddfe52.
Co-authored-by: Pedro Fontana <fontana.pedro93@gmail.com>
Contributor
Author
|
@jquesnelle @crypto-vincent @arilotter From what I see, we'd need to redeploy the program since Checkpoint now accepts the new Gcs and P2PGcs variants. But So can we remove the storage change tag? |
Collaborator
|
Yeah, i agree that it should be marked as a contract change only, and that the storage test passing is evidence that it hasn't changed in representation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement GCS Checkpoint to download model from Google Cloud Storage
Now, in the run config, we can set the model URL as a Google Cloud Storage bucket:
The GCS checkpoint is completely decoupled from Hugging Face one, because in the mid term the idea is to remove the HF checkpoint and just leave the GCS one
To access the bucket, pass the credentials as an env var:
GOOGLE_APPLICATION_CREDENTIALS=<CREDENTIAL_PATH> just start-training-localnet-light-clientAlso supports evaluation with models stored in GCS:
Summary
Checkpoint::Gcs(GcsRepo)Checkpoint::GcsP2P(gcs_repo)to falls back to GCS if P2P failspub async fn download_model_from_gcs_async()to download model from GCSIncludes