AI v.s. Human: Distinguish Machine-Generated and Human-Written Text

Welcome to our GitHub repository, where we focus on the challenge of distinguishing human-written texts from those generated by machine learning models. In this project, we apply unsupervised zero-shot methods using probability curvature to identify the nature of the text. Our approach builds on existing research by utilizing advanced models like Meta's Llama 3.1 8b-it and Google's Gemma 2 9B-it. We explore various methodologies to enhance the reliability and interpretability of text classification. Our work is based on the FollowupQ dataset, which contains human generated text and use Llama 3.1 and Gemma 2 to get the machine-generated text. We aim to provide a robust and efficient solution to the problem of detecting machine-generated text.

Use

Start by downloading the FollowupQ dataset at https://github.com/vivian-my/FollowupQG

Install the environment with the environment.yml or requirements.txt file

Put train.json, valid.json and test.json in the data map.

ENV_NAME="DL4NLP"
PYTHON_VERSION="3.10"

# Create a new Conda environment
echo "Creating Conda environment '$ENV_NAME' with Python $PYTHON_VERSION..."
conda create -y -n $ENV_NAME python=$PYTHON_VERSION

# Activate the new environment
echo "Activating the environment..."
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate $ENV_NAME

# Install the requirements from the requirements.txt file
echo "Installing requirements from requirements.txt..."
pip install -r ../requirements.txt # TODO change path of requirements
# pip install transformers torch accelerate huggingface_hub datasets

# Done
echo "Conda environment '$ENV_NAME' created and packages installed."

Run make_dataset.py on all the splits, you can use the following bash script to do so:

srun python scripts/make_dataset.py \
    --model google/gemma-2-9b-it \
    --split train \ # Change to valid or test if needed
    --folder_path /data \ # Change to the path where the data is stored
    --seed 42 \
    --token $HF_TOKEN # Add your huggingface token

We made the perturbations on the training and validation set with small_language_models.py, so you can run the following script to generate the perturbed data:

MODEL=google/gemma-2-9b-it
MODEL=meta-llama/Llama-3.1-9B-Instruct
MASK_MODEL=T5-small
MASK_MODEL=T5-large
srun python scripts/small_language_models.py \
    --file_path /home/scur1744/data/gemma/train.json \
    --hf_token HUGGINGFACE_TOKEN \
    --model_name $MODEL \
    --mask_model 'T5-small' \
    --n_perturbations 5 \
    --detector_model $DETECT_MODEL \
    --make_perturbations False \
    --denoise_with_llm False \
    --return_scores True

Run the interpretability methods implemented in interp.py with the following script:

DETECTMODEL="llama"
MODEL="llama"
HF_TOKEN=YOUR_HF_TOKEN
srun python interp.py \
    --model_name $MODEL \
    --hf_token HUGGINGFACE_TOKEN \
    --split "mini" \
    --mask_model 'T5-small' \
    --n_perturbations 5 \
    --detector_model $DETECTMODEL \
    --interp "sentence" \
    --attribution "ablation" \
    --text "Animals do not tan like humans do. They have a different skin structure that protects them from the sun's UV rays. Some animals, like bears, have thick fur that protects them, while others, like reptiles, have scales that help reflect the sun's rays. ELI5 stands for Explain Like I'm 5. It's a way to explain complex ideas in simple terms. In this case, it's asking if animals get a tan like humans do. The answer is no, but it's explained in a way that a 5-year-old can understand. ELI5 is a helpful tool for parents, teachers, and anyone who wants to explain complex ideas in a way that's easy to understand. ELI5 is a helpful tool for parents, teachers, and anyone who wants to explain complex ideas in a way that's easy to understand. It's a way to make learning fun and accessible for everyone. ELI5 is a helpful tool for parents, teachers, and anyone who wants to explain complex ideas in a way that's easy to understand. It's a way to make learning fun and accessible for everyone. ELI5 is a helpful tool for parents, teachers, and anyone who wants to explain complex ideas in a way that's easy to understand" 

Compatible Models

meta-llama/Meta-Llama-3.1-8B-Instruct

google/gemma-2-9b-it

google/gemma-2-27b-it