New Feature: Apply NLTK

[ItsRikan]

Problem

DsKit currently does not provide a high-level text data cleaning pipeline.
Although the full version installs NLTK, there is no built-in NLTK-based pre-processing utility for text/NLP workflows. This forces users to repeatedly implement custom cleaning logic outside the library.

Proposed Solution

Introduce a flexible, high-level text cleaning function named apply_nltk that enables NLP/text preprocessing directly within DsKit.

The function

• Uses NLTK internally

• Offers fine-grained control to users (case handling, stopwords, token processing, etc.)

• Is configurable and reusable across NLP pipelines

• Reduces boilerplate code for common text-cleaning tasks

• This would significantly improve DsKit’s usability for NLP and text-heavy datasets.

Current Progress

✅ Feature apply_nltk has already been implemented

✅ A Pull Request is open

🔄 Open to feedback and ready to revise:

• Code structure

• API design

• Contribution-guideline compliance

• Complexity or performance concerns

Why This Matters

• Makes DsKit more NLP-friendly out of the box

• Encourages standardized text preprocessing

• Reduces repetitive user-side implementations

• Aligns with DsKit’s goal of simplifying data preparation workflows

Related

Pull Request: #2 (comment)

[lipikaparida-web]

Hi maintainers 👋
I’m Lipika (WoC 5.0 applicant). This feature request looks very impactful for DsKit.

I noticed the apply_nltk feature already has an open PR. I’d love to contribute by helping with:

• reviewing/testing` the implementation,

• improving API design (parameters + defaults),

• adding unit tests & edge case handling, and

• writing clear usage examples/documentation.

Please let me know what part you’d prefer me to take up. I can start immediately. Thanks! 🙌

[Kritika11052005]

I’d like to take this up.

The goal would be to provide a clean, configurable high-level text preprocessing wrapper using NLTK while maintaining flexibility for users. I’ll focus on handling common edge cases (empty strings, null values, punctuation), clear parameter defaults, and strong test coverage. Refactoring will be done if needed to ensure the function remains reusable across NLP pipelines.

Looking forward to feedback before proceeding further.