File format is assumed to be packed symbols containing at most 2^20 symbols

[joshuaehill]

This isn't consistent with the format the NIST tool takes, and doesn't appear to be documented. For smaller files, the number of symbols will be adjusted, but for larger files, there appears to be no mechanism for processing more than 2^20 symbols. There shouldn't be an assumption that the number of symbols * the symbol size is divisible by 8.

[dj-on-github]

Yep. It takes packed symbols. Aka simple binary files. I’m unaware of the assumption - where is it?

…

On Tue, Sep 18, 2018 at 12:43 PM Joshua E. Hill ***@***.***> wrote: This isn't consistent with the format the NIST tool takes, and doesn't appear to be documented. For smaller files, the number of symbols will be adjusted, but for larger files, there appears to be no mechanism for processing more than 2^20 symbols. There shouldn't be an assumption that the number of symbols * the symbol size is divisible by 8. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHDBGdFUV-3AXpM1BKsmumNSDdINbYqBks5ucUzPgaJpZM4WunSN> .

[joshuaehill]

The function "read_bits_from_file" sets "symbols" to 2^20 by default, but there doesn't seem to be any way to change that without editing the code.

[joshuaehill]

Oh, unless you're referring to "There shouldn't be an assumption that the number of symbols * the symbol size is divisible by 8."
That is also in read_bits_from_file. That function always turns all input bytes into bits in the "bitlist" variable. My point is that some of those bits may not actually be associated with any symbol (in the case 8 does not divide the symbol size * the number of symbols). I haven't gotten into the test code, so it's surely possible that the trailing superfluous bits are then ignored.

[dj-on-github]

Yep. That’s a feature that isn’t there and should be. The nist tool seems to behave similarly, except it crashes a fair bit. I’ll add it in - with the proviso that if you throw a gig of data at the non-iid tests, they will never finish. DJ

…

On Tue, Sep 18, 2018 at 1:23 PM Joshua E. Hill ***@***.***> wrote: Oh, unless you're referring to "There shouldn't be an assumption that the number of symbols * the symbol size is divisible by 8." That is also in read_bits_from_file. That function always turns all input bytes into bits in the "bitlist" variable. My point is that some of those bits may not actually be associated with any symbol (in the case 8 does not divide the symbol size * the number of symbols). I haven't gotten into the test code, so it's surely possible that the trailing superfluous bits are then ignored. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHDBGeacxiTqqJl4Ub-7gYO3gi0nPBVYks5ucVZMgaJpZM4WunSN> .

[joshuaehill]

The NIST python tool assumes that 1 symbol is input per byte.

[dj-on-github]

Yes. That’s really inconvenient for production testing. It was a deliberate choice to fix that. I should put in an option to take the boneheaded NIST format.

…

On Tue, Sep 18, 2018 at 1:36 PM Joshua E. Hill ***@***.***> wrote: The NIST python tool assumes that 1 symbol is input per byte. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHDBGUpjFCYyJgJsFvVwcGL0931YYGDdks5ucVk0gaJpZM4WunSN> .

[dj-on-github]

Of course, pull requests are welcome :)

…

On Tue, Sep 18, 2018 at 1:37 PM David Johnston ***@***.***> wrote: Yes. That’s really inconvenient for production testing. It was a deliberate choice to fix that. I should put in an option to take the boneheaded NIST format. On Tue, Sep 18, 2018 at 1:36 PM Joshua E. Hill ***@***.***> wrote: > The NIST python tool assumes that 1 symbol is input per byte. > > — > You are receiving this because you commented. > > > Reply to this email directly, view it on GitHub > <#3 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AHDBGUpjFCYyJgJsFvVwcGL0931YYGDdks5ucVk0gaJpZM4WunSN> > . >

[dj-on-github]

And boy - I look grumpy in my github picture. I should fix that too.

[dj-on-github]

I've updated the main file to read in exactly the number of symbols you request via the -s switch. It will default to 1,000,000 to match the spec. I'll close this once I've added an option for the NIST oddball format.

[dj-on-github]

I added a new program bin2nistoddball at https://github.com/dj-on-github/hexbinhex that will convert a packed binary file of symbols from 1 to 8 bits per symbol and output it in the oddball format used by the NIST tool.