Context Length Issue with Qwen2.5-Coder-32B on Long Prompts

[cbacode]

Hello authors,

Thank you for releasing the code for your excellent work. I am currently attempting to reproduce the results of the paper.

However, I have encountered an issue with the maximum context length:

The prompt length for several data points exceeds 36,000 tokens. Given that Qwen2.5-Coder-32B with vllm officially supports a maximum context length of approximately 32,000 tokens, this results in silent truncation during the reproduction process.

I wonder if your team encountered this specific issue with very long prompts during your experiments. It would be incredibly helpful if you could share the strategy or technique you used to successfully handle these instances.

Thank you very much for your time and assistance.

[cbacode]

Besides, I am also confused about which dataset was used in Section 5.2 when comparing with other methods.

My confusion stems from the fact that evaluating the performance of this pipeline appears to be quite computationally expensive on the full Bird Dev set.

Could you please clarify the specific evaluation set utilized for the comparative results in that section?

[BugMaker-Boyan]

Thank you for your attention to the Alpha-SQL project. Could you provide some more detailed information? For example, which Action triggered the LLM call with a prompt content exceeding 32,000 tokens, and what is the prompt content. This will help us locate the issue (although we did not encounter this issue in our paper experiments).

For the second question you raised, we used Full BIRD-Dev and Full Spider-Dev in Section 5.2 of the paper, and for the experiments in Sections 5.3-5.5, we used the Subsampled Development Set (SDS) provided by CHESS, with the link: https://github.com/ShayanTalaei/CHESS/blob/main/data/dev/sub_sampled_bird_dev_set.json

[cbacode]

Thank you very much for your prompt response and assistance!

I apologize that I cannot immediately pinpoint the exact action or function call that specifically triggered this issue.

However, I have gathered the following crucial details regarding the context length error:

🚨 Error Observation Details
Error Range: This issue occurred repeatedly for Question IDs ranging from 1020 to 1148 during the experiment.

📜 Specific Error Message
The specific error message is consistently related to the maximum context length being exceeded, as shown below:

Question ID: 1045, Rollout step 1 / 24
----------------------------------------------------------------------------------------------------
Error calling OpenAI: Error code: 400 - {'error': {'message': "This model's maximum context length is 32768 tokens. However, your request has 37563 input tokens. Please reduce the length of the input messages. None", 'type': 'BadRequestError', 'param': None, 'code': 400}}
Start retrying 1 times
----------------------------------------------------------------------------------------------------

I hope this detailed information, especially the range of problematic IDs, is helpful for diagnosing the long-context issue. I will try my best to provide more detailed information in time, including full prompt contents in several data points.

Thank you again for your time!

[BugMaker-Boyan]

Due to the lack of detailed prompts and other information, I am sorry that I am currently unable to accurately determine the specific cause. However, fortunately, Qwen2.5-Coder-32B supports 128K context, so you can use the following command to start vllm:

# For 4 GPUs, enabling extended context up to 128K

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve Qwen/Qwen2.5-Coder-32B-Instruct --served-model-name Qwen/Qwen2.5-Coder-32B-Instruct --port 9999 -tp 4 --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}' --max-model-len 131072

[cbacode]

Sorry for the late response. Using YARN to extend the maximum model length is useful for me, but it is taking a long time for vLLM to generate a response now. The good news is that we found the detailed prompts, which are shown as follows:

qwen32b_bird_dev_exp.log

In addition, I referred to https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/bird for the evaluation code. However, I found that the result of the command bash script/sql_selection.sh has a different format from what evaluation.py requires. Should I change the format of the output myself?

[BugMaker-Boyan]

Thank you for providing the logs; they were instrumental in identifying the issue: The Database Schema's Value Examples contain excessively long values, causing the Prompt Length to rapidly increase. The specific cause lies in the absence of length restrictions or filters on values at this line:

load_value_examples.

We will address this promptly. Your insights are greatly appreciated.

To quickly resolve the current issue for you, we recommend changing Return Line to [example[0] for example in examples if len(str(example[0])) < 100].

[BugMaker-Boyan]

Of course, if you have the time, you can submit a PR to fix this problem, and we are honored to have you as one of the contributors to Alpha-SQL😄.

[cbacode]

Thank you for your quick response.

I am honored to be given the chance to contribute to this project. I will put in my best effort to address and fix this issue, although I cannot promise a perfect solution immediately.

[cbacode]

Sorry for late response. I changed the code like

res = [example[0] for example in examples]
    print(res, flush=True)
    if str(res) > MAX_LENGTH_COLUMN_DESCRIPTION:
        return [res[0]]
    else:
        return res

instead of return [example[0] for example in examples] in the exact same place in function load_value_examples, but I could not notice any change in my log. So I am sorry that I am not quite sure this could work.

[cbacode]

I have created a merge request and feel free to change it.