How to use the ds_channel_config_path parameter when using SGLang?

[BigCousin-z]

When using SGLang to perform inference on DeepSeek's R1 large model, the ds_channel_config_path parameter needs to be provided by DoubleSparse. However, I haven't found out how to generate this configuration file.

[andy-yang-1]

@BigCousin-z Hi, you can get the config file by adding support for this code. Since DeepSeek R1 is MLA architecture, I think I need to add extra support for this. It is better to try with DeepSeek-Qwen first which is GQA.

[BigCousin-z]

@BigCousin-z Hi, you can get the config file by adding support for this code. Since DeepSeek R1 is MLA architecture, I think I need to add extra support for this. It is better to try with DeepSeek-Qwen first which is GQA.

I get an error when trying to load the DeepSeek R1 model,：
root@gpuserver:~DoubleSparse# python3 config/offline_calibration.py --model_pat /media/modelhub/deepseek_r1 --output_dir .
The repository for /media/modelhub/deepseek_r1 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co//media/modelhub/deepseek_r1.
You can avoid this prompt in future by passing the argument trust_remote_code=True.

Do you wish to run the custom code? [y/N] Traceback (most recent call last):
File "/home/miniconda3/envs/sglang-nsa/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 651, in resolve_trust_remote_code
answer = input(
File "/home/miniconda3/envs/sglang-nsa/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 632, in _raise_timeout_error
raise ValueError(
ValueError: Loading this model requires you to execute custom code contained in the model repository on your local machine. Please set the option trust_remote_code=True to permit loading of this model.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/DoubleSparse/config/offline_calibration.py", line 175, in
model = AutoModelForCausalLM.from_pretrained(model_path, **kwargs)
File "/home/miniconda3/envs/sglang-nsa/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "/home/miniconda3/envs/sglang-nsa/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1009, in from_pretrained
trust_remote_code = resolve_trust_remote_code(
File "/home/miniconda3/envs/sglang-nsa/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 664, in resolve_trust_remote_code
raise ValueError(
ValueError: The repository for /media/modelhub/deepseek_r1 contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co//media/modelhub/deepseek_r1.
Please pass the argument trust_remote_code=True to allow custom code to be run.

and there is no such parameter “trust_remote_code=True”.

[BigCousin-z]

any process？

[andy-yang-1]

@BigCousin-z I think this is your huggingface transformers' setting problem. It is not related to Double Sparsity

[jifa513]

@BigCousin-z Hi, you can get the config file by adding support for this code. Since DeepSeek R1 is MLA architecture, I think I need to add extra support for this. It is better to try with DeepSeek-Qwen first which is GQA.

I am generating the offline calibration file for Qwen/Qwen3-Coder-30B-A3B-Instruct, and I get an error with the offline_calibration.py :

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:20<00:00, 1.29s/it]
Collecting activation scales...
Repo card metadata block was not found. Setting CardData to empty.
WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty.

• Split into 124 blocks
  0%| | 0/124 [00:00<?, ?it/s]
  Traceback (most recent call last):
  File "/sgl-workspace/jifa/DoubleSparse/config/offline_calibration.py", line 183, in
  output_dict = get_calib_feat(model, tokenizer)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
  return func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/jifa/DoubleSparse/config/offline_calibration.py", line 161, in get_calib_feat
  model(input_ids)
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
  return forward_call(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py", line 175, in new_forward
  output = module._old_forward(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 959, in wrapper
  output = func(self, *args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py", line 651, in forward
  outputs: MoeModelOutputWithPast = self.model(
  ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
  return forward_call(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 1083, in wrapper
  outputs = func(self, *args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py", line 482, in forward
  hidden_states = decoder_layer(
  ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py", line 94, in call
  return super().call(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
  return forward_call(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py", line 175, in new_forward
  output = module._old_forward(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py", line 342, in forward
  hidden_states, _ = self.self_attn(
  ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
  return forward_call(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py", line 175, in new_forward
  output = module._old_forward(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py", line 162, in forward
  query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1879, in _call_impl
  return inner()
  ^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1840, in inner
  hook_result = hook(self, args, result)
  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/jifa/DoubleSparse/config/offline_calibration.py", line 53, in get_q_hook
  assert y.shape[-1] == model.config.hidden_size
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  AssertionError