Add Ask-LLM data curation example

[xyuzh]

Summary

This PR adds the Ask-LLM data curation example demonstrating LLM-based data curation using the Ask-LLM methodology from the DCLM paper.

What it does

• Uses Qwen2.5-3B-Instruct via vLLM to score text quality
• Implements the Ask-LLM approach: prompts an LLM with "Is this text suitable for training a language model?"
• Uses the softmax probability of "Yes" (P(Yes)) as a quality score
• Filters the FineWeb-edu dataset based on quality threshold
• Writes curated data to parquet

Files

• main.py: Main pipeline with preprocessing, LLM inference, and postprocessing
• job.yaml: Anyscale job configuration (8x g5.xlarge GPUs)
• Dockerfile: Container with vLLM and dependencies
• README.md: Documentation and usage instructions

References

• DCLM paper
• How to Train Data-Efficient LLMs