Operator Precedence bug in dns.c

[nigeltao]

In https://github.com/jbangert/nail/blob/master/examples/dns/dns.c there is this snippet:

pos = highbyte & 63 << 8 | current->data[pos];

or

pos = highbyte & 63 << 8 | etc;

In C, the operator precedence is <<, then &, then |, so that the equivalent parenthesized expression is:

pos = (highbyte & (63 << 8)) | etc;

Folding the 63<<8 constant gives:

pos = (highbyte & 0x3F00) | etc;

highbyte is a uint8_t, obviously in the range [0x00, 0xFF], so and'ing it with 0x3F00 will always be zero:

pos = 0 | etc;

which simplifies to:

pos = etc;

which is:

pos = current->data[pos];

which is clearly an error, as it ignores the highbyte value. Presumably the original statement should have been:

pos = (highbyte & 63) << 8 | current->data[pos];

although it's not obvious if that is still semantically correct, for compressed DNS records.

[pictyeye]

We just ran into the same problem (which led to CPU-intensive never-ending loops) and we believe adding the parenthesis as in the last snippet is the correct way to fix this.

pos = (highbyte & 63) << 8 | current->data[pos];

(line 54 should also be changed similarly)

We also have found other issues with the way DNS compression is handled in dns.c, so we may propose a PR soon.