Access to LM ARPA Files and Training Text Data

[csetanmayjain]

Could we also get access to the LM ARPA files and the underlying text data?
Currently, the shared language model is only available in binary format, which limits visibility into its structure.

Having access to the ARPA/text data would help us:

• Understanding the tokenizer structure
• Analyzing spoken punctuation handling
• Extending or adapting the LM
• Performing LM interpolation with other datasets

Thanks

[srikantak-221]

Hi @csetanmayjain,

Thank you for reaching out and providing this details. We have escalated this issue to our Engineering team to look into this further. We are tracking this internally and will provide an update on this thread as soon as we have a resolution or next steps.

[csetanmayjain]

Hi team,
Following up on this, could you please share any updates?
Also, it would be helpful to know the expected timeline for a response.