bABI Task 17

[navdeepkjohal]

Dear Researcher,

I hope this message finds you well.

I have a question regarding bAbI Task 17 and the generation of ASP facts.

When prompting the LLM to generate ASP facts for Task 17, the output typically looks like this:

obj(1, (pink, rectangle)).
obj(2, (triangle)).
obj(3, (red, square)).
leftOf(1,2).
leftOf(2,3).

However, the ASP rules provided for bAbI Task 17 in the repository (linked below) appear to assume a different representation:

LLM-ASP/asp_domains.py

Line 144 in 642a9ad

task17 = '''

Specifically, the rules expect facts of the following form:

obj(1).
color(1, pink).
shape(1, rectangle).

obj(2).
shape(2, triangle).

obj(3).
color(3, red).
shape(3, square).

leftOf(1,2).
leftOf(2,3).
leftOf_nondirect(1,3).

Could you please clarify whether there is any internal preprocessing or transformation step that converts the LLM-generated obj/2 predicates into the corresponding color/2 and shape/2 predicates? If so, I would appreciate any guidance on where this logic is implemented or how it is intended to work.

Thank you very much for your time and help.

[AdamIshay]

Hi navdeepkjohal,

Sorry for the late reply. Task 17 uses a custom function which uses the LLM to parse: https://github.com/azreasoners/LLM-ASP/blob/main/bAbI/babi_utils.py#L116.

In this function, first, facts like the following are parsed using the LLM and the story (without the query sentence):

obj(1, (pink, rectangle)). obj(2, (triangle)). obj(3, (red, square)).
Direction Facts: leftOf(1,2). leftOf(2,3).

From this, only the direction facts will be placed in the ASP program. The facts like obj(1, (pink, rectangle)) are only used as context to the LLM when further generating the query fact (e.g., rightOf_nondirect(1,3)), which will also be placed in the ASP program.

So the facts like obj(1, (pink, rectangle)) can be thought of as helping the LLM to keep track of the object indexes (e.g., the pink rectangle is object 1) when generating the query fact which uses them.