Proposal: Python download tool

[knghtbrd]

Currently we are using get in our she'll scripts to download files and not really doing much validation on the files thus downloaded. We've also got a few loop constructs in she'll scripts for downloading a file from two or potentially more sources.

If we had our own tool, we could get a file from a list of specified mirrors, its expected file name, an expected size, a file hash, etc. A command line specifying all of that would be unwieldy (and I'd argue it might be a good argument for a here document in a script), but it'd be one command that would just get you the file you want and verify its integrity, or return an error if it can't.

I propose this tool be written in Python because most of the code to write it already exists in my Python experimental tree, save for some shell command friendly glue logic. Also, if written that way, the tool can be used both standalone from a shell script or by just importing it internally into other Python modules which already makes our more complex (and ugly) scripts more approachable to those who don't have as much regents talents. ;)

Other argument: Most A2 stuff seems to use Java to be platform-independent. Java can be used everywhere, but it doesn't exactly play nice anywhere. Python does play nice, even on Windows, and doesn't need wrapper scripts. It also doesn't force procedural tools into an OO modality like it or not, but has the OO when you want it. Plus it can actually do a proper CLI app and native or native-enough GUI widgets. If AppleCommander, ADTPro, and all the other Java-based tools out there today became Python-based, it wouldn't be a bad thing! Of course where Python falls down over alternatives is interfacing C libraries, but that doesn't affect us here. :)

[IvanExpert]

I am all for Python over Java, so by all means, do what you gotta.

On Dec 21, 2015, at 4:55 AM, Joseph Carter notifications@github.com wrote:

Currently we are using get in our she'll scripts to download files and not really doing much validation on the files thus downloaded. We've also got a few loop constructs in she'll scripts for downloading a file from two or potentially more sources.

If we had our own tool, we could get a file from a list of specified mirrors, its expected file name, an expected size, a file hash, etc. A command line specifying all of that would be unwieldy (and I'd argue it might be a good argument for a here document in a script), but it'd be one command that would just get you the file you want and verify its integrity, or return an error if it can't.

I propose this tool be written in Python because most of the code to write it already exists in my Python experimental tree, save for some shell command friendly glue logic. Also, if written that way, the tool can be used both standalone from a shell script or by just importing it internally into other Python modules which already makes our more complex (and ugly) scripts more approachable to those who don't have as much regents talents. ;)

Other argument: Most A2 stuff seems to use Java to be platform-independent. Java can be used everywhere, but it doesn't exactly play nice anywhere. Python does play nice, even on Windows, and doesn't need wrapper scripts. It also doesn't force procedural tools into an OO modality like it or not, but has the OO when you want it. Plus it can actually do a proper CLI app and native or native-enough GUI widgets. If AppleCommander, ADTPro, and all the other Java-based tools out there today became Python-based, it wouldn't be a bad thing! Of course where Python falls down over alternatives is interfacing C libraries, but that doesn't affect us here. :)

—
Reply to this email directly or view it on GitHub #52.

[knghtbrd]

If you have any experience writing (or even looking at) Python scripts, you might be able to help with this. The idea I have in mind is a tool that runs from a shell script, but can also be imported from a Python program and used that way.

It seems that a useful tool for this should know the file you're trying to download, a list of URLs to get it from, the name you want the file written to, and optionally some verification support. Size, hash, etc. The module should probably throw exceptions if something doesn't go right, and the command line tool should probably catch those and issue numeric return codes.

Do the mirror URLs include the filename or is that a separate argument? If the former, can there be more than one kind of file depending on where it is? Apple has some stuff in BinHex format that isn't compressed like that on Asimov for example. Is that worth dealing with? Feel free to discuss. :)