Added i) directory support, ii) FP rate args, iii) No-Save option

[revbucket]

Several changes to main.rs:

1. Added progress bar printouts vs printouts at each filename (tried to use similar formatting as in wimbd)
2. Added directory support for inputs (can pass dir as input now, only selects .json*.gz files). Same as wimbd
3. Added option to not save bloom filter at the end (but update during)
4. Added option to compute filter size dynamically given a desired false-positive rate

[revbucket]

Oh and some more notes about point 4:
We search for a size of bloom filter using an accurate ngram count and then find the smallest filter that results in that fp_rate. We return min(filter_size, 0.90 * systemRAM) (so we don't allocate too much memory -- maybe I should warn if we get to this point though!)

[dirkgr]

What happens with the progress bar if the output of bff is piped into a file? Or the terminal is in a weird state?

This is why I generally prefer sticking to writing to stdout and stderr, one line at a time, and that's all. It causes the least amount of trouble in unexpected settings.

[dirkgr]

I have one suggestion. If that's too hard, let's ship it like this.

[dirkgr]

Extra newline?

[revbucket]

😅
Yes will fix this before shipping

[dirkgr]

Can Arc pointers be null? Then you could save a parameter and pass in null here instead.

[revbucket]

Yes...? Pretty sure we can wrap with an option and then we don't have to pass the second parameter