Negated char class : sacrifice 1 or 2 tokens to improve range merge

[Siorki]

There is a possible optimisation in the Packer, when merging character ranges to build the negated char class.

If the last token only improves the compression by 1 byte (token gain = 1), and that removing one extra character would merge two ranges (gain 3), then not performing that compression step would free the token and actually improve the final result by 2 bytes.

The algorithm should therefore :

• identify the leaf tokens (those with no further dependency on them) with the least gain
• when merging range, measure the gain given by one extra token.

If the gain from the merge exceeds the one from the token, remove the token, merge further, and perform another check.

[Siorki]

Experiments with the prototype have been inconclusive. So far, it brought no improvement on any of the files in the benchmark.

On the plus side, it demonstrated that sacrificing a token can shorten the negated char class. There are several examples from the benchmarks with gains of 3 bytes for one token.
Autumn Evening brings this further by having [a-ik-pr-y] in the char class , meaning that both j and q can be sacrificed for 3 bytes each.

On the minus side, the sacrifices are usually not worth it.
After reordering patterns from the crusher to account for dependencies, only the leaves of the tree graph can be sacrificed. Those leaves are the shortest strings since the dependency A->B means "B contains A". Unlike the crusher phase where the early replacements are worth a lot and the final ones only 1 or 2 bytes, the gain is more evenly spread in the packer.
Which unfortunately means that the leaves are usually worth 3 or 4 bytes, and there will be no gain in sacrificing them, even for a gain of 3.

Putting the issue on hold until a gain can be obtained from a real example.