This project focuses on counterspeech (and hateful speech) detection in online discussions by developing a text classifier using both traditional ML methods and LLMs. The datasets used consists of 2399 (counter speech) and 14,000 (hateful speech) German comments from a mock social media platform (datasets available upon request), manually labeled as counterspeech (1) or not counterspeech (0) by three human coders. Initially, text preprocessing techniques were applied, followed by various feature extraction methods including BoW, TF-IDF, and word embeddings (i.e. Doc2Vec, BERT, and OpenAI’s text-embedding-ada-002). Multiple ML classifiers (logistic regression, SVM, decision trees, random forests) were trained, incorporating SMOTE and random oversampling to address class imbalance.
However, these models struggled with recall on counterspeech instances, leading to the exploration of LLM-based zero-shot and few-shot classification using GPT-4. The LLM-based approach was implemented by generating a classification prompt in German, instructing GPT-4 to act as a research assistant (following Macanovic & Przepiorka, 2024) and classify each short comment with a binary label. Initially, a zero-shot classification method was used, resulting in higher recall than ML models but low precision. To improve accuracy, a few-shot classification approach was introduced, providing examples of counterspeech and non-counterspeech in the prompt. Batch processing strategies were optimized to respect OpenAI's rate limits, ensuring efficient token usage.
data_preprocessing.R: This script handles data preprocessing in R, including text cleaning, tokenization, lowercasing, punctuation removal, and stopword filtering. It also prepares the dataset for further analysis by extracting comment lengths, performing initial exploratory data analysis, and exporting the cleaned dataset for use in ML models. Additionally, it includes feature extraction (BoW, TF-IDF, word embeddings) methods. If you are viewing the data_preprocessing.R file in RStudio, it is highly recommended clicking through the collapsible sections (i.e. the section headers) to get a clearer overview of the workflow.
ml_classification.ipynb: This (IPython) Jupyter Notebook script implements ML classification methods to detect counterspeech and hateful speech. It includes data balancing techniques (SMOTE, random oversampling) and ML algorithms (Logistic Regression, SVM, Decision Trees, Random Forests). Additionally, the script explores different regularization parameters (L1, L2), class weighting, and hyperparameter tuning, while also varying the train/test split proportions and applying stratification to maintain class distribution. Finally, it evaluates model performance using accuracy, precision, recall, F1-score, and confusion matrices to compare different classification approaches. Although the counterspeech classifier struggles to achieve high precision (when identifying counterspeech comments), the hateful speech classifier performs substantially better, correctly identifying over 70% of hateful instances, in line with similar studies in the field using German data. Notably, the Support Vector Machine (SVM) model using TF-IDF unigrams yielded the best performance, achieving a strong balance of precision, recall, and accuracy.