Welcome to the ChatGPT Benchmarking Framework! This project aspires to be the ultimate testing ground for ChatGPT, evaluating its prowess in solving coding challenges and problems from various domains. Dive in and see the potential of where we want to take ChatGPT's benchmarking!
- Comprehensive Benchmarking: Aim to assess ChatGPT's capabilities across a wide range of computational tasks and challenges.
- Adaptable to Evolving AI: As AI models grow and evolve, this framework will scale and adapt to remain the gold standard of evaluation.
- Collaborative Refinement: With the community's involvement, we envision refining and expanding the range and depth of tests.
This is the BEGINNER'S EDITION of the framework! It's an initial draft, capturing the essence of our grand vision. While it's packed with budding features, it's not the final masterpiece. Your insights and contributions are invaluable in our journey!
- Problem Extraction: Extracts problems from a variety of file formats.
- Solution Evaluation: Checks the solutions provided by ChatGPT against set tests.
- Scoring Mechanism: Provides an overall score and categorical scores based on ChatGPT's performance.
- JSON Result Export: Exports results in a structured JSON format for analysis.
- Setup the environment by installing the required packages and setting up the OpenAI API key.
- Position your problem files and corresponding test files in the specified directories.
- Activate the benchmarking script.
- Watch the magic as ChatGPT dives into your challenges!
This is the dawn of our journey! We're keen to incorporate your feedback, suggestions, and innovations. Have a groundbreaking challenge for ChatGPT? Share it! Discover a bug or visualize an enhancement? Let's craft the future together!
This project is open-source and under the embrace of the MIT License.