Observation

[AymanBx]

fairnessBench/fairnessBench/agents/agent_research.py

Lines 233 to 278 in 5d28be7

	def summarize_observation(self, action, observation, log_file, bs = 10000):
	""" Summarize the observation if it is too long with a sliding window of size bs """
	bs = 10000
	blocks = [observation[i:i+bs] for i in range(0, len(observation), bs)]
	descriptions = []
	for idx, b in enumerate(blocks):
	start_line_number = bs*idx+1
	end_line_number = bs*idx+1 + len(b)
	prompt = f"""
	{action}
	The full observation is too long. Given this (partial) observation from character {start_line_number} to character {end_line_number}:
	```
	{b}
	```
	Summarize the observation concisely in this format:
	[Observation]: Summarize all relevant details in the observation objectively
	Do not include any result that is guessed rather than directly confirmed by the observation. Do not include additional information or suggestions.
	"""
	completion = complete_text_fast(prompt, log_file=log_file +f"_{idx}")
	descriptions.append(completion)
	if len(descriptions) == 1:
	completion = descriptions[0]
	else:
	descriptions = "\n\n".join(["Segment {idx}: \n\n" + s for s in descriptions])
	prompt = f"""
	{action}
	The full observation is too long.
	Given summaries for each segments of the whole observation, summarize to get a cohesive description of the entire observation.
	{descriptions}
	Summarize the observation concisely in this format:
	[Observation]: Summarize all relevant details in the observation objectively
	Do not include any result that is guessed rather than directly confirmed by the observation. Do not include additional information or suggestions.
	"""
	completion = complete_text_fast(prompt, log_file=log_file)
	try:
	return completion.split("[Observation]:")[1]
	except:
	return completion

If we use "Observation" in the stop sequence then we lose summarization. We can either change "Observation" to "Summary" in the prompt here or remove the word "Observation" from the stop sequence.

Similarly, the word "Observation" is used in the retrieval step prompt... (The next function)

[brownsarahm]

is "observation" in any of the research problems causing an issue?

[AymanBx]

It might; having it in the prompt rases the chances of the word being generated in the response, and if the LLM generates the word in the response then the response gets cut off.