Undefined reference

[jankokondic]

I'm trying to use the daulet/tokenizers library in a Go program via cgo, but I'm encountering an issue when running the program — I get errors saying that functions like from_file, encode, etc. are undefined ("undefined reference").

What I did:

• Cloned the tokenizers repository.

• Ran make build, which successfully generated the static library libtokenizers.a.

• Wrote a minimal Go program that calls functions from the library.

• Tried to run the program with the following command:

go run -ldflags="-extldflags '-L./'" .

The libtokenizers.a file is located in the current directory.

The problem:
During the build process, I get linker errors indicating that functions such as from_file, encode, etc. cannot be found.

I'd really appreciate any help or advice on how to correctly link this library with a Go program.

Thanks in advance!

[Inquisitor-201]

The version of the Go library (tokenizers) you're using might be outdated, while the libtokenizers.a library compiled with Rust is newer, leading to compatibility issues. Try running go mod tidy, then check the version of the Go tokenizers library in your go.mod file. Next, switch this repository to the corresponding version and recompile.

[daulet]

What @Inquisitor-201 said.

@WatchJani function names have been recently renamed so you just need to ensure you are using consistent version between Go library and the built libtokenizers.a

[arkadijs]

For people reaching here via search: make sure there is no other (incompatible) version of tokenizers in your go.work 🙈

[daulet]

FWIW the recent versions add a better link time error specifying which version of the lib your go library is looking for.