zminio

this is a minio client package that allows you to these actions with your minio server.

1 ) upload : upload files to your minio server. ( every file you want! )
2 ) uploadDir : upload a directory full of multiple files to the minio.
3 ) download : download one object or all of the objects from one bucket in minio server.
4 ) delete : delete one object or all of the objects from one bucket in minio server.
5 ) list : get the list of objects from one bucket in minio server.
6 ) info : get the information of one object from minio server.
7 ) move : move one object from one bucket to another bucket in minio server.
8 ) sync : sync two bucket from minio server with each other.
9 ) listenDownload : listen to the one bucket in minio server , and download every object that will be uploaded.
10 ) sideloader : it will split large files into a specefic amount of size and transfer those base on the scp or tls connection.

Note

you can set the config of the minio servers from flags and .env file.

Note

you can set your prometheus server address for monitor your proccess with this flag -pr.

Note

The template of the grafana panel is in docker folder with name zminio-grafana-panel.json.

Note

This app is suppporting concurrency, and you can set the amount of workers with -n flag. (default = 10)

Flags

      --ak string         set your minio access key
      --aks string        set your minio access key
      --all               sync all the buckets and objects
      --bs string         set your minio bucket name
  -b, --bucket string     set your minio bucket name
      --cache             use the redis as a cache for the sync proccess!
      --do string         set the job you want to do. (download, upload, move, delete, list, sync, uploadDir)
      --ds                delete the object from bucket after sync
  -u, --endpoint string   set your minio url address
      --env string        set your env file path
  -h, --help              help for zminio
  -f, --input string      set the path of the file that you wanna upload
  -i, --interval int      set the interval for sync objects (default 1)
  -l, --logger string     set app logger type , stdout or file (default "stdout")
      --ls                set the listen bucket on sync proccess!
      --mb string         set your minio bucket name that you want to move files to it
      --mincache          use the minio as a cache for the sync proccess!
      --obj string        set your minio object name
  -o, --output string     set the path of the file that you wanna download
      --pr string         run Prometheus on ip:port to monitor aminio metrics,if not set this flag prometheus disabled. (exaple:-pr 0.0.0.0:1234)
      --save              save the objects throw the sync process!
      --scp               use scp command for download sideloader chuncks
      --se                set your secure ssl option in connecting to the minio. (default false)
      --ses               set your secure ssl option in connecting to the minio. (default false)
      --sideloader        sideloader for transfer large objects over tcp connection.
      --sk string         set your minio secret key
      --sks string        set your minio secret key
      --storage string    set your minio storage path for get the list of objects.
      --tag string        set your object tag for get the list and download
  -t, --type string       set the side loader type. (server, client)
      --us string         set your minio url address
  -v, --version           zminio version
  -n, --workers int       set the count of worker for run. (default 10)

Sample Commands

NOTE : you have to create .env file for set minio login data or you could pass them via the provided flags.

• upload :

./Zminio -f README.md --do upload --env zminio.env

• upload directory :

./Zminio -f /tmp/uploads/ --do uploadDir --env zminio.env -n 50

NOTE : you can set the number of workers to upload objects to the minio. (base on the source you have)

• download :

./Zminio --obj README.md -o /tmp/Downloads/ --do download --env zminio.env

• download all :

./Zminio --obj all -o /tmp/Downloads/ --do download --env zminio.env -n 50

• download listen bucket :

./Zminio -o /tmp/Downloads/ --do listenDownload --env zminio.env -n 10

NOTE : By default zminio in listenDownload senario, does not delete object after download. For do that you can use this flag --ds to delete objects after download.

NOTE : You can use --tag for download just specific objects with tag. (if you want to download specific tag and value use this = --tag "key":"value" and if you want to download just a tag key with any value use this = --tag "key":*)

• delete :

./Zminio --obj [object name] --do delete --env zminio.env

• delete all :

./Zminio --obj all --do delete --env zminio.env -n 50

• move :

./Zminio --do move --obj [object name] --mb MOVE_BUCKET --env zminio.env

• sync :

./Zminio --do sync --env zminio.env --ds -n 50

[!NOTE]
--ds delete objects after sync minio's together.
for save objects into the storage while sync in happening, you can use --save and set the output path for download!
after sync will be finished, the app will be close. For ignore that and listen on the bucket to sync comming objects you have to use --ls flag.
if you want to sync all buckets and objects, you have to use --all flag.
in case of sync , its better to have a cahce for sync process to ignore exsited file in the destination. For doing that you should set the cache address in .env file and use --cache flag.

• list :

./Zminio --do list --env zminio.env

NOTE: if you use --storage the app will calculate the count of objects from the minio storage path.

NOTE : You can use --tag for get list of specific objects with tag. (if you want to get list of specific tag and value use this = --tag "key":"value" and if you want to get list of tag key with any value use this = --tag "key":*)

• info :

./Zminio --do info --obj [object name] --env zminio.env

Sideloader

this app will be able to create a socket between server and client over tcp, then it will communicate and transfer large files base on a protocol like FTP.

• server command :

./Zminio --sideloader -t server --env zminio.env

• client command :

./Zminio --sideloader -t client --env zminio.env ....

NOTE: if you set --scp flag to the command it will transfer files with scp command, however you can transfer files without using --scp command which it will happen by default and it does not need any flag or config.

ENV file

the format of .env file should be like this:

MINIO_ENDPOINT="localhost:9000"
MINIO_ACCESS_KEY="minioadmin"
MINIO_SECRET_KEY="minioadmin"
MINIO_BUCKET_NAME="test"
MINIO_SSL_SECRET="true"
SYNC_MINIO_ENDPOINT="localhost:9000"
SYNC_MINIO_ACCESS_KEY="minioadmin"
SYNC_MINIO_SECRET_KEY="minioadmin"
SYNC_MINIO_BUCKET_NAME="test"
SYNC_MINIO_SSL_SECRET="true"
REDIS_BACKEND_URL='rediss://:password@localhost:6379'
SIDELOADER_ADDRESS='0.0.0.0:3030'
SIDELOADER_SIZE='250'
SIDELOADER_MAXIMUM_SIZE_START='100'
SIDELOADER_SERVER_ADDRESS='localhost:3030'
SIDELOADER_CERT='./server-cert.pem'
SIDELOADER_KEY='./server-key.pem'
SCP_ADDRESS=''
SCP_PORT=''
SCP_USERNAME=''
SCP_PASSWORD=''

Note

SYNC_MINIO_ENDPOINT is address that you want to copy your objects.

Zminio Test Result

we made some tests to know about the performance of this module on the internet and local networks. We gather all results in this table :

NETWORK	Total Objects Size	Network Speed	Executed Time	Service Scenario
LOCAL	12.5Gib	100Mib	16m	scp sync scenario
LOCAL	12.5Gib	100Mib	21m	file transfer sync scenario
LOCAL	12Gib	100Mib	1:30m	just zminio file transfer
INTERNET	250Mib	17 Mbps	4:08m	just zminio file transfer
INTERNET	250Mib	17 Mbps	3:58m	just scp file transfer