Duplicate Article Detection and Clustering

[mircealungu]

Problem: Multiple articles covering the same news event lead to redundant recommendations, especially problematic when users mark content as uninteresting.

Proposed Solution:

• Use embeddings or maybe bad hashing to detect semantically similar articles describing the same event (then maybe use an LLM to fine-tune the detection?)
• Implement clustering to group related articles together
• Select the highest-quality article from each cluster for recommendation

Optional, more challenging path

• Cluster various articles on the same topic and let the user know that they are related
• Show the learner all the possible related sources for a given topic (as google news does)

Implementation Considerations:

• Embedding-based similarity detection is cost-effective compared to LLM pairwise comparisons
• Need threshold tuning to balance over-clustering vs under-clustering
• Consider temporal factors (articles published within similar timeframes about the same topic)