Skip to content

Automatically upload big files using DVC #295

@simonlsk

Description

@simonlsk

When uploading a new directory with the CLI

dagshub upload <repo> <local-dir-path> <remote-dir--path>

The directory is uploaded using DVC.
When uploading a single file using the same command, the file is always uploaded with git.
It would be nice to have a size threshold (i.e 5MB) that would automatically decide to upload the file using DVC.

The interesting question is how do you prevent the repo from growing into a list of many single dvc tracked files, and make sure the user makes use of dvc directories to store big files in a manner that makes sense:

.
├── data  <-- dvc
│   ├── preprocessed
│   │   └──  003.png <-- single file
│   └── raw
├── models <-- dvc
└── src <-- git

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions