v1.25.0

What's Changed

• feat: add WASM support by @daulet in #53
• chore: organize the repo for multi crates by @daulet in #54
• fix: pages build by @daulet in #55
• feat: build release artifacts in ci by @daulet in #57

Full Changelog: v1.24.0...v1.25.0

v1.24.0

What's Changed

• feat: add s390x and ppc64le architectures by @sheltoncyril in #48

New Contributors

• @sheltoncyril made their first contribution in #48

Full Changelog: v1.23.0...v1.24.0

v1.23.0

What's Changed

• Update HuggingFace tokenizers dependency to v0.22.0 by @Nav31 in #44
• catch and return panics in encoders by @steved in #46
• bump tiktoken-rs/fancy-regex to 0.16 by @steved in #45

New Contributors

• @Nav31 made their first contribution in #44
• @steved made their first contribution in #46

Full Changelog: v1.22.2...v1.23.0

v1.22.2 fix: handle decoding partial UTF-8 characters

Full Changelog: v1.22.1...v1.22.2

v1.22.1 Link time version check

Full Changelog: v1.22.0...v1.22.1

v1.22.0 Better errors on Tokenizer init

Full Changelog: v1.21.1...v1.22.0

v1.21.1 Tiktoken stability

Full Changelog: v1.21.0...v1.21.1

v1.21.0 Tiktoken support

Full Changelog: v2.20.2...v1.21.0

v1.20.2

What's Changed

• feat: better error message when tokenizers lib mismatch by @daulet in #28
• feat: FromPretrained to load tokenizer directly from HF by @berkayersoyy in #27

New Contributors

• @berkayersoyy made their first contribution in #27

Full Changelog: v0.9.0...v1.20.2

v0.9.0

What's Changed

• feat: add option to retrieve offsets from tokenizer by @riccardopinosio in #21
• Update to huggingface/tokenizers v0.20.0 by @daulet in #23

New Contributors

• @riccardopinosio made their first contribution in #21

Full Changelog: v0.8.0...v0.9.0