A learning project that reimplements core features of GNU wget in Rust. The goal is to understand networking, concurrency, filesystem handling, and website mirroring using a compiled, memory-safe language.
This is not a full replacement for wget.
- Download a file from a URL
- Save a file with a custom name
- Save a file to a specific directory
- Show download progress and statistics
- Limit download speed
- Download in background with logging
- Download multiple files asynchronously
- Mirror an entire website for offline use
- HTTP
- HTTPS (FTP optional, depending on implementation)
- Rust (stable)
- Cargo
Run directly with Cargo:
$ cargo run -- https://pbs.twimg.com/media/EMtmPFLWkAA8CIS.jpgOr build a binary:
$ cargo build --release
$ ./target/release/wget-clone https://example.com/file.zipBasic download:
$ cargo run -- https://pbs.twimg.com/media/EMtmPFLWkAA8CIS.jpgExample output:
start at 2017-10-14 03:46:06
sending request, awaiting response... status 200 OK
content size: 56370 [~0.06MB]
saving file to: ./EMtmPFLWkAA8CIS.jpg
55.05 KiB / 55.05 KiB [==============================================] 100.00% 1.24 MiB/s 0s
Downloaded [https://pbs.twimg.com/media/EMtmPFLWkAA8CIS.jpg]
finished at 2017-10-14 03:46:07
If the response status is not 200 OK, the program exits with an error.
Runs the download in the background and redirects output to wget-log.
$ cargo run -- -B https://pbs.twimg.com/media/EMtmPFLWkAA8CIS.jpg
Output will be written to "wget-log".Save the file under a different name.
$ cargo run -- -O=meme.jpg https://pbs.twimg.com/media/EMtmPFLWkAA8CIS.jpgSave the file to a specific directory.
$ cargo run -- -P=~/Downloads/ -O=meme.jpg https://pbs.twimg.com/media/EMtmPFLWkAA8CIS.jpgLimit download speed.
Supported units:
k→ kilobytes per secondM→ megabytes per second
$ cargo run -- --rate-limit=400k https://example.com/file.zip
$ cargo run -- --rate-limit=2M https://example.com/file.zipDownload multiple files asynchronously from a text file.
$ cat download.txt
https://assets.01-edu.org/wgetDataSamples/20MB.zip
https://assets.01-edu.org/wgetDataSamples/Image_10MB.zip
$ cargo run -- -i=download.txtAll downloads run concurrently.
Mirror an entire website locally.
$ cargo run -- --mirror https://example.comFiles are stored in a directory named after the domain:
www.example.com/
Skip downloading files with specific extensions.
$ cargo run -- --mirror -R=jpg,gif https://example.comSkip specific paths while mirroring.
$ cargo run -- --mirror -X=/assets,/css https://example.comRewrite links so the mirrored site works offline.
$ cargo run -- --mirror --convert-links https://example.comThis project focuses on:
- Rust networking (blocking or async)
- Streaming downloads
- Async/concurrency (
tokio, threads, or channels) - Rate limiting
- Progress reporting
- Recursive crawling
- HTML parsing
- Safe filesystem operations
- CLI design
- Error handling is explicit and strict
- Designed for Unix-like systems
- Output format intentionally mimics wget
- Features may be incomplete or evolving