This repository presents our solution for the Detect AI-generated Text Competition, a national-level data science competition hosted in 2025, Dacon.
The goal of the cahllenge was to detect whether a paragraph was written by a Large Language Model (LLM), with only document-level labels provided. This required re-egineering the data, designing weak supervision strategies, and buildig robust classifiers in a highly imbalanced setting.
- Host: Dacon
- Track: Detect AI-generated Text
- Evaluation Metric: ROC-AUC
- Input: Document-level 'full_text' (Provided as train.csv)
- Label: 'generated' (0 for human-written, 1 for AI-generated)
- Goal: Classify each paragraph as human- or AI-generated
| title | full_text | generated |
|---|---|---|
| 카호올라웨섬 | 카호올라웨섬은 하와이 제도를 구성하는 (중략...) | 0 |
| 청색거성 | 천문학에서 청색거성은 광도 분류 (중략...) | 0 |
| 수난곡 | 수난곡은 배우의 연기 없이 무대에 (중략...) | 1 |
| ID | title | paragraph_index | paragraph_text |
|---|---|---|---|
| TEST_0000 | 공중 도덕의 의의와 필요성 | 0 | 도덕이란 원래 개인의 자각... |
| TEST_0001 | 공중 도덕의 의의와 필요성 | 1 | 도덕은 단순히 개인의 문제... |
| TEST_0002 | 공중 도덕의 의의와 필요성 | 2 | 여기에 이른바 공중도덕은... |
| ID | generated |
|---|---|
| TEST_0000 | 0 |
| TEST_0001 | 0 |
Our core challenge was to reconstruct paragraph-level labels from document-level data in an extremely imbalanced and noisy setting. To overcome this, we developed multiple weak supervision and filtering strategies, focusing on data-centric approaches.
- Each
full_text(up to 9 paragraphs) was split into multipleparagraph_textunits - For long paragraphs, we applied sliding window chunking with overlapping stride
- Input format:
"제목: {title} 본문: {paragraph_text}"
The dataset exhibited severe class imbalance:
approximately 10:1 ratio between generated=0 and generated=1 labels.
To address this, we performed both data augmentation and filtering for high-confidence positive samples.
- Oversampling + Label Propagation
- Filtering using Perplexity and Semantic Similarity
We used a pretrained KoGPT model ("kanana") to generate synthetic paragraphs mimicking generated=1 style.
These augmented paragraphs were added to the training set to enrich the positive class.
- Model: kakaocorp/kanana-1.5-2.1b-instruct-2505 (instruction-tuned KoGPT variant)
- Prompt-based generation: Input includes preceding ([BEFORE]) and following ([AFTER]) paragraphs. The model generates a coherent middle ([TARGET]) paragraph mimicking generated=1 style.
- Post-processing:Used kss for sentence segmentation. Removed incomplete trailing sentences using regex-based ending pattern matching (다, 요, 습니다 etc.). Applied manual spot-checking to remove low-quality generations. -Labeling & Integration: All generated paragraphs labeled as generated=1. Added to the training set to enrich the positive class distribution.
- Scale: Generated 11,317 synthetic paragraphs for augmentation.
🔁 4) Weak Labeling & Filtering Strategies learn more->
We experimented with various heuristic and unsupervised labeling techniques:
- Used GPT-like models to compute perplexity scores
- Sentences or paragraphs with low perplexity were considered likely AI-generated
- Applied thresholding to extract
generated=1candidates
- Extracted syntactic style features (e.g., average sentence length, token diversity)
- Combined with perplexity
- Performed KMeans or HDBSCAN clustering to isolate AI-like patterns
- Applied sentence-wise perplexity estimation
- Clustered using HDBSCAN to extract dense positive clusters
- Label assigned to entire paragraph if one or more sentences clustered as AI
- Modified (perturbed) the sentence input
- Measured change in log-likelihood (LL) or model confidence
- Used as a proxy for model “surprise” or fluency robustness
The first baseline model used KLUE-RoBERTa (Which you can find and use in hugging face). The model with the best performance in STS.
| Model | STS | TC |
|---|---|---|
| mBERT-base | 84.66 | 81.55 |
| XML-R-base | 89.16 | 83.52 |
| KLUE-RoBERTa-base | 92.50 | 85.07 |
After that, many attempts were made using many models. leand more ->
- Sliding window over tokens with max pooling
- Sentence-level Pertubation + Perplexity filtering
- KLUE-RoBERTa embedding + cosine similarity for pseudo-labeling
- Model ensemble using ranking + voting
Our final submission used an ensemble of three models, trained on a paragraph-level relabeled dataset
1. KLUE-RoBERTa-large (fine-tuned)
- Chosen for its strong semantic representation capability, which is crucial for detecting subtle contextual inconsistencies in AI-generated text
- Check training details here
2. KLUE-RoBERTa-base (fine-tuned)
- Selected as a lighter and more generalizable counterpart to the large model, reducing overfitting risk on imbalanced data
- Check training details here
3. CatBoost Classifier
- Focused on stylometric anomalies often observed in AI-generated text, complementing semantic models
- Features: unique_ratio, verb_ratio, entropy, polynomial interaction terms (degree=2), Perplexity (PPL) estimated via
skt/kogpt2-base-v2 - Check training details here
A custom Extreme Voting method was adopted to maximize ROC-AUC:
- ≥2 models ≥0.5 → max(probabilities) (optimistic consensus)
- Otherwise → min(probabilities) (conservative fallback)
Our team of 5 members divided responsibilities as follows:
Team Name: PokSSak(폭싹)
| 소속 | 이름 | 역할 | |
|---|---|---|---|
| 숙명여자대학교 컴퓨터사이언스전공(22) | 김소영 | Data Engineering, Relabeling & Model Development | soyoung2118 |
| 숙명여자대학교 데이터사이언스전공(23) | 김수빈 | ||
| 숙명여자대학교 데이터사이언스전공(23) | 오현서 | Data Engineering & Experiment Execution | |
| 숙명여자대학교 데이터사이언스전공(23) | 원지우 | Data Engineering & Experiment Execution | |
| 숙명여자대학교 소프트웨어융합전공(22) | 임소정 | Data Engineering & Model Development | sophia |
We collaborated for 4 weeks.
Environment
Development
Communication
- Final submission score: ROC-AUC 0.889
- Full competition leaderboard: [https://dacon.io/competitions/official/236473/leaderboard]
- PDF summary: 'docs/project_summary.pdf'