Conceptual Combination with Property Type (CCPT)

📖 CCPT arXiv | 🤗 CCPT

This is the official repository of our NAACL 2025 main (Oral) paper: Is Peeled Apple Still Red? Evaluating LLM for Conceptual Combination with Property Type

Have any questions?

Please contact at seokwon.song@vision.snu.ac.kr

Citation

If you use CCPT in your research, please cite our work:

@article{song2025peeled,
  title={Is a Peeled Apple Still Red? Evaluating LLMs' Ability for Conceptual Combination with Property Type},
  author={Song, Seokwon and Lee, Taehyun and Ahn, Jaewoo and Sung, Jae Hyuk and Kim, Gunhee},
  journal={arXiv preprint arXiv:2502.06086},
  year={2025}
}

Installation

First, clone our GitHub repository.

git clone https://github.com/seokwon99/CCPT.git

Then navigate to the newly-created folder.

cd CCPT

Next, create a new Python 3.9+ environment using conda.

conda create --name ccpt python=3.9

Activate the newly-created environment.

conda activate ccpt

All external package requirements are listed in requirements.txt. To install all packages, and run the following command.

pip install -r requirements.txt
pip install flash-attn --no-build-isolation
python -m spacy download en_core_web_sm

Reimplementation

Set Environment

export OPENAI_API_KEY="sk-..."    # if you want openai models
export ANTHROPIC_API_KEY="sk-..." # if you want anthropic models

Get Model Response

python -m experiment.run.gen_property --property_type emergent    # property induction (emergent)
python -m experiment.run.gen_property --property_type canceled    # property induction (canceled)
python -m experiment.run.gen_combination --property_type emergent # noun phrase completion (emergent)
python -m experiment.run.cls_property_type                        # property type prediction

Automatic Evaluation

python -m experiment.eval.eval_property --property_type emergent    # property induction (emergent)
python -m experiment.eval.eval_property --property_type canceled    # property induction (canceled)
python -m experiment.eval.eval_combination --property_type emergent # noun phrase completion (emergent)
python -m experiment.eval.eval_type                                 # property type prediction