Guidance regarding Topic modeling experiment

[krishnakanthnakkav2]

Hi Team,

Thank you for maintaining such a wonderful repository.

I have a question regarding the topic analysis using BERTopic mentioned at the bottom of your HuggingFace page (https://huggingface.co/spaces/LLM360/TxT360).

Could you please confirm if the 17 category names (e.g., Arts, Business & Economics, Finance, etc.) are predefined by your team? If not, could you guide me on the pipeline how to obtain this list of category names?

Thanks,
Krishna

[hunterhector]

Hi, @krishnakanthnakkav2 ,

Thanks for interested in our work.

Here is a rough process (since I didn't work on it myself some of the details may vary, but the gist is here):

To get to the categories, we first use a small document set (probably around 10K) to try out a few clustering settings, creating about 128 cluster automatically. And then we manually look at the clusters, and try to merge them into something ~20 groups. This number is simply chosen to reduce the number of clusters we can analyze. We then label them with the help of manual inspection and GPT prompting.

After that, we calculate an embedding vector of the each cluster, and check the similarities between every document with this vector to assign them into the clusters.