Skip to content

Gpt 5 fix #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bluescreen10 merged 2 commits into from
Aug 8, 2025
Merged

Gpt 5 fix #21

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5cc8c3c
Update tokenizer.go
ando-masaki Aug 8, 2025
3ac0d0b
Merge pull request #2 from ando-masaki/ando-masaki-patch-1
ando-masaki Aug 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion tokenizer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ package tokenizer
// - O1
// - O1Preview
// - O1Mini
// - GPT5
// - GPT5Mini
// - GPT5Nano
// - GPT4.1
// - GPT4o
// - GPT4
Expand Down Expand Up @@ -107,6 +110,9 @@ const (
O3 Model = "o3"
O3Mini Model = "o3-mini"
O4Mini Model = "o4-mini"
GPT5 Model = "gpt-5"
GPT5Mini Model = "gpt-5-mini"
GPT5Nano Model = "gpt-5-nano"
GPT41 Model = "gpt-4.1"
GPT4o Model = "gpt-4o"
GPT4 Model = "gpt-4"
Expand Down Expand Up @@ -199,7 +205,7 @@ func Get(encoding Encoding) (Codec, error) {
// is returned.
func ForModel(model Model) (Codec, error) {
switch model {
case O1, O1Preview, O1Mini, GPT41, GPT4o, O3, O3Mini, O4Mini:
case O1, O1Preview, O1Mini, GPT5, GPT5Mini, GPT5Nano, GPT41, GPT4o, O3, O3Mini, O4Mini:
return Get(O200kBase)

case GPT4, GPT35, GPT35Turbo, TextEmbeddingAda002:
Expand Down