benchmark eval changes

[danielchalef]

• benchmarks
• Add LongMemEval dataset analysis scripts and .gitignore files
• refactor
• refactor and cleanup
• Refactor dataset analysis scripts for improved readability and consistency
• Add start index parameter to LongMemEvalRunner for controlled data ingestion
• wip

---

Important

This pull request adds scripts for evaluating Locomo and LongMemEval datasets using Zep's knowledge graph and OpenAI's LLM, including ingestion, response generation, grading, and analysis functionalities.

• Locomo Evaluation:
  • Adds zep_locomo_eval.py for grading responses using OpenAI's LLM with a custom grading prompt.
  • Adds zep_locomo_ingestion.py for ingesting Locomo dataset into Zep's knowledge graph.
  • Adds zep_locomo_responses.py for generating responses using OpenAI's LLM based on Zep's context.
  • Adds zep_locomo_search.py for searching and composing context from Zep's knowledge graph.
• LongMemEval Evaluation:
  • Adds zep_eval.py and zep_responses.py for grading and generating responses using OpenAI's LLM.
  • Adds analyze_dataset_sessions.py for analyzing session statistics from LongMemEval dataset.
  • Adds calculate_scores.py for calculating scores by question type from JSONL results.
  • Adds queue_depth.py for analyzing queue metrics from a JSON file.
  • Adds zep_longmem_eval.py for a comprehensive evaluation pipeline including ingestion, response generation, and grading using Zep and OpenAI's LLM.

^{This description was created by for c0c0196. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

Caution

Changes requested ❌

Reviewed everything up to a008fed in 2 minutes and 34 seconds. Click for details.

• Reviewed 292 lines of code in 3 files
• Skipped 0 files when reviewing.
• Skipped posting 4 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. benchmarks/longmemeval/queue_depth.py:4

• Draft comment:
  Consider wrapping execution logic in a main() guard and adding error handling for file I/O.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% This is a new file in a benchmarks directory, likely meant for ad-hoc analysis. While error handling and main() guards are good practices, this seems like overkill for a simple benchmark script. The comment is technically correct but may not add value given the context and purpose of the file. The comment suggests good engineering practices that could make the code more robust. File I/O can fail and having proper error handling is generally good practice. Given this is a benchmark script in a benchmarks directory, it's likely meant to be simple and straightforward. Adding error handling would make it more verbose without adding much value in this specific context. The comment should be removed as it suggests overengineering a simple benchmark script. While technically correct, it violates the rule about not making comments unless there is clearly a code change required.

2. benchmarks/longmemeval/zep_longmem_eval.py:163

• Draft comment:
  The tar.extractall() call uses filter="data", but filter expects a callable. Verify the intended usage for safe extraction.
• Reason this comment was not posted:
  Comment was not on a location in the diff, so it can't be submitted as a review comment.

3. benchmarks/longmemeval/zep_longmem_eval.py:392

• Draft comment:
  Ensure that DatasetManager, ZepIngester, and LLMEvaluator are imported or defined, as they are used in the init method.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 0% vs. threshold = 50% This is a legitimate issue - the code is trying to use classes that don't exist in the current file. This would cause runtime errors. The comment is pointing out a real problem that needs to be fixed. However, the phrasing "Ensure that..." makes it a verification request rather than a direct statement of the issue. The comment uses "Ensure that..." phrasing which violates our rules about verification requests. It could be more direct in stating the issue. While the phrasing could be better, the underlying issue is real and important - the code will not work without these missing classes being defined or imported. The comment identifies a critical issue that needs to be fixed, but should be rephrased to directly state the problem rather than asking for verification.

4. benchmarks/longmemeval/zep_longmem_eval.py:499

• Draft comment:
  Method naming inconsistency: The grader method was renamed to 'grade_response' but is still called as 'lme_grader'. Update the call to match the new method name.
• Reason this comment was not posted:
  Comment was not on a location in the diff, so it can't be submitted as a review comment.

Workflow ID: wflow_exPxn4y2dvPa9J7C

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[ellipsis-dev]

Replace the print() call with a logger call (e.g., logger.debug/info) for consistent logging.

Suggested change

	print(f"Question max: {question_max}")
	self.logger.info(f"Question max: {question_max}")

[ellipsis-dev]

Caution

Changes requested ❌

Reviewed 89703c3 in 2 minutes and 35 seconds. Click for details.

• Reviewed 117 lines of code in 3 files
• Skipped 0 files when reviewing.
• Skipped posting 6 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. benchmarks/longmemeval/calculate_scores.py:55

• Draft comment:
  Add a trailing newline at the end of the file to comply with POSIX standards.
• Reason this comment was not posted:
  Confidence changes required: 33% <= threshold 50% None

2. benchmarks/longmemeval/queue_depth.py:6

• Draft comment:
  Guard against missing keys in the JSON by checking if the key exists before accessing it.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 30% vs. threshold = 50% This is a benchmark/evaluation script, not production code. The key seems to be a hardcoded UUID, suggesting this is for analyzing specific known data. Error handling might make the code more robust, but it's not clear if it's necessary for this use case. The comment doesn't provide strong evidence that missing keys are actually a problem here. The comment raises a valid defensive programming practice. Missing key handling could prevent crashes if the data format changes. While defensive programming is good practice, this appears to be a benchmark script for specific known data. Over-engineering error handling may not be valuable here. Delete the comment as it suggests adding complexity that may not be necessary for this benchmark script's specific use case.

3. benchmarks/longmemeval/queue_depth.py:15

• Draft comment:
  Check if 'gwf' is non-empty before dividing to avoid a ZeroDivisionError.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% The code assumes the key "GraphWorkflow-1fe79d02-2163-4b3b-8530-677551b358c6" exists and contains data. If the key doesn't exist, it would raise a KeyError before reaching the division. If the key exists but maps to an empty dict, a ZeroDivisionError could occur. However, this seems like a simple metrics script where the author knows their data structure. The comment identifies a real potential error case. Empty dictionaries could cause runtime errors that aren't immediately obvious. However, this appears to be a one-off metrics script working with known data. The hardcoded UUID suggests this is not meant to be robust production code. The comment, while technically correct, is overly defensive for a metrics script working with known data structure. It violates the rule about not making speculative comments.

4. benchmarks/longmemeval/zep_longmem_eval.py:33

• Draft comment:
  Consider using 'datetime' consistently instead of splitting it as 'date time' in the context template for clarity.
• Reason this comment was not posted:
  Confidence changes required: 33% <= threshold 50% None

5. benchmarks/longmemeval/zep_longmem_eval.py:164

• Draft comment:
  The 'filter' argument in tar.extractall must be a callable, not a string. Update or remove it to correctly filter archive members.
• Reason this comment was not posted:
  Comment was not on a location in the diff, so it can't be submitted as a review comment.

6. benchmarks/longmemeval/calculate_scores.py:55

• Draft comment:
  It appears there's no newline at the end of the file. Adding a trailing newline would adhere to common style guidelines.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% While having a newline at the end of file is a common convention and some tools expect it, this is a very minor style issue. It's something that could be automatically fixed by an autoformatter or linter. The comment doesn't point out a functional issue or suggest an important code quality improvement. Missing newlines at end of file can cause issues with some Unix tools and make diffs less clean. It's a real issue that some would consider worth fixing. While true, this is exactly the kind of minor style issue that should be handled by automation rather than code review comments. It doesn't affect functionality and isn't worth the reviewer's or author's time. Delete the comment as it points out a very minor style issue that would be better handled by automation.

Workflow ID: wflow_EFBWQlFDAHEwXhcy

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[ellipsis-dev]

Important

Looks good to me! 👍

Reviewed c0c0196 in 1 minute and 11 seconds. Click for details.

• Reviewed 13 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. benchmarks/longmemeval/zep_longmem_eval.py:34

• Draft comment:
  Grammar fix: Changed 'date time' to 'datetime' for clarity.
• Reason this comment was not posted:
  Comment looked like it was already resolved.

Workflow ID: wflow_sw93zeiQsSC16uJF

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}