To reduce the amount of dependencies (or rather concentrate more on dfdx), the Tokenizer (token encoding/decoding) should be ported to dfdx. This also applies to the token processor, which makes a token selection based on the Vocab output logits.
To reduce the amount of dependencies (or rather concentrate more on dfdx), the Tokenizer (token encoding/decoding) should be ported to dfdx.
This also applies to the token processor, which makes a token selection based on the Vocab output logits.