Big OpenLM/DCLM <-> AI2 PR # 1#12

Open

revbucket wants to merge 67 commits intoallenai:mainfrom

Commits on Feb 29, 2024

Barebones python/S3 implementation

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 5, 2024

Bash script written

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 7, 2024

Added i) directory support, ii) FP rate args, iii) No-Save option

Matt Jordan
authored and
Matt Jordan
committed
Made changes requseted in PR

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 8, 2024

Added --no-progress flag

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 13, 2024

Added options for no-progress (cleaner signature) | Updated README.md to describe some new features

Matt Jordan
authored and
Matt Jordan
committed
Cleaned up readme duplicates

Matt Jordan
authored and
Matt Jordan
committed
Merged main

Matt Jordan
authored and
Matt Jordan
committed
Bash script fixed

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 14, 2024

first commit

Matt Jordan
authored and
Matt Jordan
committed
Added fp rate to scripts

Matt Jordan
authored and
Matt Jordan
committed
added no-save-bloom-filter option in bash script by default

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 15, 2024

Hacky AWS script seems to work okay

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 17, 2024

s3 stream stuff added

Matt Jordan
authored and
Matt Jordan
committed
Oops, forgot to add threading

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 18, 2024

Subcommands for local + s3 stuff built

Matt Jordan
authored and
Matt Jordan
committed
Added sysreq command

Matt Jordan
authored and
Matt Jordan
committed
Okay, overrode the sysreq blockers

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 19, 2024

Added better i/o collecting for bff-remote

Matt Jordan
authored and
Matt Jordan
committed
Added streaming support for s3-in

Matt Jordan
authored and
Matt Jordan
committed
Added default region for AWS

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 20, 2024

Added some more timing/ablations for some various checks

Matt Jordan
authored and
Matt Jordan
committed
More testing stuff

Matt Jordan
authored and
Matt Jordan
committed
Removing debuggy things

Matt Jordan
authored and
Matt Jordan
committed
Support subset
Ubuntu
committed
Added some better printouts for removal stats at the end

Matt Jordan
authored and
Matt Jordan
committed
Merge pull request #1 from achalddave/s3-stream
revbucket
authored

Commits on Mar 21, 2024

Added retry support for bff-remote

Matt Jordan
authored and
Matt Jordan
committed
Added better RT thread control?

Matt Jordan
authored and
Matt Jordan
committed
Merged, should have {retries, better RT threading}

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 22, 2024

Commits on Mar 25, 2024

Freezing status for dedup ablations v3

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 27, 2024

Freezing status for dedup ablations v3

Matt Jordan
authored and
achalddave
committed
Added i) parallel init for bits vector; ii) better output directory formatting, iii) better whitespace/empty document filtration

Matt Jordan
authored and
Matt Jordan
committed

Commits on Mar 29, 2024

Support offset
achalddave
committed
Merge pull request #2 from achalddave/s3-stream
revbucket
authored
Printed number of input files

Matt Jordan
authored and
Matt Jordan
committed
Mreged achal's pr

Matt Jordan
authored and
Matt Jordan
committed
Added sharding control for BFF, printed out filter sparsity at the end

Matt Jordan
authored and
Matt Jordan
committed

Commits on Apr 8, 2024

Allowing text in annotate mode

Matt Jordan
authored and
Matt Jordan
committed
idk, cargo needed updating

Matt Jordan
authored and
Matt Jordan
committed
Added scripts/ to gitignore

Matt Jordan
authored and
Matt Jordan
committed
Merge branch 'dabv3'

Matt Jordan
authored and
Matt Jordan
committed
Update README.md
revbucket
authored
Cleaned things up a bit

Matt Jordan
authored and
Matt Jordan
committed
Merge branch 'main' of github.com:revbucket/bff

Matt Jordan
authored and
Matt Jordan
committed
Made bloom-filter-file not mandatory

Matt Jordan
authored and
Matt Jordan
committed
merged

Matt Jordan
authored and
Matt Jordan
committed

Commits on Apr 14, 2024

Commits on Apr 15, 2024

Folded in no-stream for bff-remote

Matt Jordan
authored and
Matt Jordan
committed
Merge branch 'no-stream'

Matt Jordan
authored and
Matt Jordan
committed
Added s5cmd wrapper

Matt Jordan
authored and
Matt Jordan
committed
oops

Matt Jordan
authored and
Matt Jordan
committed

Commits on Apr 17, 2024

Ready for PR; @dirk, it's time to d-d-duel

Matt Jordan
authored and
Matt Jordan
committed