question about qwen3 finetune

[Hannibal046]

Hi, thanks for the quick support for the qwen3-embedding model!

I noticed a discrepancy regarding the query prefix format. In the official post, the instruction and query are concatenated without a space. However, in the Tevatron example, there's a space after the colon:

  --query_prefix "Find a relevant scientific paper abstract to support or reject the claim. Query: " \

This seemingly minor difference actually affects tokenization behavior:

# Without space after colon
>>> print(tokenizer.tokenize("Answer the query\nQuery:What is the capital of China?"))
['Answer', 'Ġthe', 'Ġquery', 'Ċ', 'Query', ':', 'What', 'Ġis', 'Ġthe', 'Ġcapital', 'Ġof', 'ĠChina', '?']

# With space after colon
>>> print(tokenizer.tokenize("Answer the query\nQuery: What is the capital of China?"))
['Answer', 'Ġthe', 'Ġquery', 'Ċ', 'Query', ':', 'ĠWhat', 'Ġis', 'Ġthe', 'Ġcapital', 'Ġof', 'ĠChina', '?']

[ArvinZhuang]

Hi Xin Cheng! @Hannibal046

You are right that the space will result in little difference, and we should probably follow the QWEN official post and remove that space in the tevatron example.

Also, I notice in the official post there is an instruct prefix, I think the correct usage is:
--query_prefix "Instruct: Find a relevant scientific paper abstract to support or reject the claim.\nQuery:"

By saying that, I find it odd that they have a space between Instruct: and the task, but no space for the query. I hope they are correct about their official post :)

[ArvinZhuang]

Hi @Hannibal046, I just did an inference test and updated the qwen3 example.

I found there were a few other mismatched details.

We usually do padding_side = right for Tevatron models, but it seems qwen3 requires padding side left. In addition, we often do --append_eos_token in tevatron, which adds a eos token at the end of all texts, but it seems qwen3 doesn't do this.

Now our example should match the official qwen3 encoding, but please verify on you side as well :)

[ArvinZhuang]

@Hannibal046
just want to clarify one thing about adding padding token at the end of texts.
Qwen3 embedding tokenizer will append <|endoftext|> special token to every text, which is the token used to pool the text embedding.

--append_eos_token in tevatron will additionally append a tokenizer.eos_token which is <|im_end|> for qwen3 tokenizer to every texts, thus we should not use --append_eos_token