Add missing HF functionalities

[benITo47]

While trying to migrate ReactNative-Executorch from hf-tokenizers cpp bindings to your implementation, I found several issues, mostly with parsing and applying tokeniser.json config.

This PR mitigates some of them:

• Prepend normaliser is handled
• PreTokenizer explicitly set to null is handled
• ByteFallback flag is respected
• Added skip_special_tokens flag to decode function
• Added piece_to_id public method
• Fixed some CPP issues

[benITo47]

cc: @larryliu0820 @mergennachin

[mergennachin]

I don't like that this change is a breaking change. One we update the tokenizer pin in ExecuTorch, it will break all the call sites

What is your thought of doing something like this?

  class Tokenizer {
   public:
    // New batch API (pure virtual)
    virtual Result<std::string> decode(
        const std::vector<uint64_t>& tokens,
        bool skip_special_tokens = false) const = 0;

    // Old streaming API - non-breaking, delegates to batch
    // Subclasses can override if they need prev_token context
    virtual Result<std::string> decode(uint64_t prev_token, uint64_t token) const {
      (void)prev_token;  // Default: ignore prev_token like BPE does
      return decode({token}, false);
    }
  };

  // SPTokenizer overrides to preserve prev_token behavior
  class SPTokenizer : public Tokenizer {
   public:
    Result<std::string> decode(uint64_t prev_token, uint64_t token) const override {
       ...
    }
  };

[benITo47]

I don't like that this change is a breaking change.

I agree. Honestly, at first I only GREPed executorch repository for CPPHFTokenizer, and found no issues.
I reverted those changes, now batch decoding is available as overload only in the hf_tokenizer, similar to your snippet.

Please take a look again :-)

Also, I have a custom test suite, that I was using to test whether tokenizers for models we provide in rn-execeutorch are working as expected. Would you be interested in getting this suite? It's kinda .json heavy and I would need to strip a lot of content so I don't push 500k lines. Let me know.

[mergennachin]

See inline, also add tests for skip_special_tokens and piece_to_id logic.

[mergennachin]

There's some syntax error in pybind, according to the unit test failure.

[mergennachin]

Great, @benITo47

I have a few more requests. See inline

[mergennachin]

Is it an Error or could it be downgrade to Info or Debug?

[mergennachin]

Thank you @benITo47