Use functor to generate the hash modules, add MD5 digest.

[madroach]

Hi,

here are some big changes. I trimmed down the stub code to the minimum algorithmic code and imlemented as much in OCaml as possible. To reduce redundancy I put the code in a functor which takes the stubs as parameter and generates the hash modules (Sha1, MD5...). Then I added the MD5 digest using the C functions already present in the OCaml runtime.
I also fixed a bug in commit fe6dba9.

I could take maintainership of the project, but would like to move it to BSD or ISC license.

Christopher

[madroach]

I obviosly would agree licensing my commits to ISC or BSD.

[talex5]

@djs55 this is a big PR, but I think we should cherry-pick the memory-corruption fix fe6dba9 at least. I've made a separate PR #36 for this.

[djs55]

Thanks for this -- I completely approve of the goal of implementing as much in OCaml as possible!

Sorry for the conflicts -- would you be able to rebase this on top of current master?

[madroach]

This is not finished yet. I still may have to merge some changes to the imlementation files that happened since I converted them to a single functor

[madroach]

Ok. I believe this is ready to me merged. The changes to the repo since I opened the pull requested were already included in my functor.
Still unfinished is the file_fast function. I would leave it as is and remove the C stub code. If this isn't fast enough we could switch from channels to Unix.read.

[madroach]

I added back the file_fast function and benchmarked it against an unbuffered implementation with more code in the functor. The results for the very small sample.txt is below. Please note that the benchmark is run on thousands of times on the very same file so the file is buffered by the OS.

                    Rate                  file file_unbuffered       file_fast
           file  28795+- 542/s              --            -85%            -89%
file_unbuffered 192040+-5202/s            567%              --            -27%
      file_fast 261630+-7906/s            809%             36%              --

addendum - benchmark 610kb file, too:

                 Rate    file_unbuffered       file_fast            file
file_unbuffered 299+-3/s              --           [-0%]             -4%
      file_fast 299+-4/s            [0%]              --             -4%
           file 311+-3/s              4%              4%              --

this tells me the real performance gain of file_fast and file_unbuffered comes from the overhead of Unix.openfile and open_in.
Since any implementation is ridiculously fast on small files and there is no difference for big files I would suggest dropping the file_fast and file_unbuffered stubs. Then we can also drop the dependency on Unix.

[XVilka]

Was anything decided about this?