Conversation
Add a detailed README.md that includes: - Project overview and description - List of supported emulators (17 total) - Test ROM suites documentation - Installation and requirements - Usage examples with CLI flags - Output format explanation - Project structure overview Closes gbdev#40
Add a complete README.md covering: - Project overview and purpose - List of supported emulators with links - Test suite descriptions - Installation and usage instructions - Project structure explanation - Contributing guidelines - License information Closes gbdev#40
ISSOtm
left a comment
ISSOtm
left a comment
There was a problem hiding this comment.
Hey! Thanks for submitting this.
The style of what you submitted doesn't match our project's spirit, but that's no big deal, the substance is a good thing to start discussing off of. I have a bunch of requests below, but if you have pushback, feel free to state your reasons aloud!
| ## Supported Emulators | ||
|
|
||
| The framework currently supports testing the following emulators: | ||
|
|
||
| | Emulator | URL | | ||
| |----------|-----| | ||
| | [Beaten Dying Moon](https://mattcurrie.com/bdm-demo/) | mattcurrie.com/bdm-demo | | ||
| | [mGBA](https://mgba.io/) | mgba.io | | ||
| | [KiGB](http://kigb.emuunlim.com/) | kigb.emuunlim.com | | ||
| | [SameBoy](https://sameboy.github.io/) | sameboy.github.io | | ||
| | [bgb](https://bgb.bircd.org/) | bgb.bircd.org | | ||
| | [VisualBoyAdvance](https://sourceforge.net/projects/vba) | sourceforge.net/projects/vba | | ||
| | [VisualBoyAdvance-M](https://github.com/visualboyadvance-m/visualboyadvance-m) | github.com/visualboyadvance-m | | ||
| | [No$gmb](https://problemkaputt.de/gmb.htm) | problemkaputt.de/gmb | | ||
| | [GambatteSpeedrun](https://github.com/pokemon-speedrunning/gambatte-speedrun) | github.com/pokemon-speedrunning/gambatte-speedrun | | ||
| | [Emulicious](https://emulicious.net/) | emulicious.net | | ||
| | [Goomba](https://www.dwedit.org/gba/goombacolor.php) | dwedit.org/gba/goombacolor | | ||
| | [binjgb](https://github.com/binji/binjgb) | github.com/binji/binjgb | | ||
| | [PyBoy](https://github.com/Baekalfen/PyBoy) | github.com/Baekalfen/PyBoy | | ||
| | [ares](https://ares-emu.net/) | ares-emu.net | | ||
| | [Emmy](https://emmy.n1ark.com/) | emmy.n1ark.com | | ||
| | [gameroy](https://github.com/Rodrigodd/gameroy) | github.com/Rodrigodd/gameroy | | ||
| | [DocBoy](https://github.com/Docheinstein/docboy) | github.com/Docheinstein/docboy | | ||
|
|
There was a problem hiding this comment.
The list of emulators is already present in the website itself, so this list would easily fall out of sync and thus become problematic; I'd rather it be absent from the README. Perhaps the README's introduction line can evoke the variety of listed emulators?
| ## Supported Emulators | |
| The framework currently supports testing the following emulators: | |
| | Emulator | URL | | |
| |----------|-----| | |
| | [Beaten Dying Moon](https://mattcurrie.com/bdm-demo/) | mattcurrie.com/bdm-demo | | |
| | [mGBA](https://mgba.io/) | mgba.io | | |
| | [KiGB](http://kigb.emuunlim.com/) | kigb.emuunlim.com | | |
| | [SameBoy](https://sameboy.github.io/) | sameboy.github.io | | |
| | [bgb](https://bgb.bircd.org/) | bgb.bircd.org | | |
| | [VisualBoyAdvance](https://sourceforge.net/projects/vba) | sourceforge.net/projects/vba | | |
| | [VisualBoyAdvance-M](https://github.com/visualboyadvance-m/visualboyadvance-m) | github.com/visualboyadvance-m | | |
| | [No$gmb](https://problemkaputt.de/gmb.htm) | problemkaputt.de/gmb | | |
| | [GambatteSpeedrun](https://github.com/pokemon-speedrunning/gambatte-speedrun) | github.com/pokemon-speedrunning/gambatte-speedrun | | |
| | [Emulicious](https://emulicious.net/) | emulicious.net | | |
| | [Goomba](https://www.dwedit.org/gba/goombacolor.php) | dwedit.org/gba/goombacolor | | |
| | [binjgb](https://github.com/binji/binjgb) | github.com/binji/binjgb | | |
| | [PyBoy](https://github.com/Baekalfen/PyBoy) | github.com/Baekalfen/PyBoy | | |
| | [ares](https://ares-emu.net/) | ares-emu.net | | |
| | [Emmy](https://emmy.n1ark.com/) | emmy.n1ark.com | | |
| | [gameroy](https://github.com/Rodrigodd/gameroy) | github.com/Rodrigodd/gameroy | | |
| | [DocBoy](https://github.com/Docheinstein/docboy) | github.com/Docheinstein/docboy | |
There was a problem hiding this comment.
The website doesn't really present a convenient "list" of emulators; you have to scroll sideways over a large table and read its column headers. Short lists of the tested emulators and of the test suites are two things I'd want to see in a README.
There was a problem hiding this comment.
That said, it's weird for the table-style "list" to have visible URLs that aren't linked. Linking them would be redundant with the titles, so just remove the visible URLs; anyone can easily hover/click on the titles to see the pages.
There was a problem hiding this comment.
This list is not short, however, it's comprehensive, which makes it redundant in my eyes. I don't want to try making a shortlist unless we have some criteria to base them off of; and even those may be subject to constant bikeshedding.
As for the website presenting a list of emulators, I have in mind that this should be solved by implementing a filter for emulators, which would incidentally serve as such a list that doesn't require horizontal scrolling. (And such filtering would remove the horizontal scrolling, too!)
There was a problem hiding this comment.
I'm not sure what you mean by a "shortlist" with "some criteria to base them off of".
My use of "large" meant "you have to scroll horizontally a lot to read all the emulators' table headings, which is bad UX". (It's inevitable for a table, of course, but means that users should not be expected to treat the table headings as a readable list on their own.)
And my use of "short list" was just an adjective, like "short enough to fit on one screen without scrolling"; no implication that it would be shortened by some criteria to not include every single emulator we test.
There was a problem hiding this comment.
It is short enough to fit one a single screen, even a mobile one; however, it is long enough and lacking obvious organisation that I don't see what a user is supposed to glean from it.
Further, as more emulators are going to be added, this list is going to grow beyond that “short” qualifier; IMO it's already questionably short.
|  | ||
|
|
There was a problem hiding this comment.
Hmm, I don't think the license is particularly relevant to this project, since it's not a distributable; I'd rather have it only mentioned at the bottom.
|  |
There was a problem hiding this comment.
We can also just rely on the fact that a LICENSE file exists and is detected by GitHub to present in its repo sidebar. If we really want to mention the license in the README, I think it should be done as plain text.
| A comprehensive comparison framework for testing Game Boy emulator accuracy across hundreds of test ROMs. | ||
|
|
||
| ## Overview | ||
|
|
There was a problem hiding this comment.
This is redundant with the sentence below, so I'd like to keep only one. And, personally, I find the latter more descriptive of the project :)
| A comprehensive comparison framework for testing Game Boy emulator accuracy across hundreds of test ROMs. | |
| ## Overview |
|
|
||
| ## Overview | ||
|
|
||
| GBEmulatorShootout automatically tests multiple Game Boy emulators against a suite of accuracy test ROMs, generating detailed HTML reports with pass/fail status and screenshots for each test case. |
There was a problem hiding this comment.
| GBEmulatorShootout automatically tests multiple Game Boy emulators against a suite of accuracy test ROMs, generating detailed HTML reports with pass/fail status and screenshots for each test case. | |
| GBEmulatorShootout tests multiple Game Boy emulators against a suite of accuracy test ROMs, compiling all the pass/fail results into a comprehensive table, with screenshots for each test case. | |
| The table can be generated locally, but we also regularly update it at https://gbdev.io/GBEmulatorShootout/ |
Stylistic tweaks, and placing a link to the site prominently in the README itself.
There was a problem hiding this comment.
s/statuses/results/ and s/obtained/generated/, but yes.
| - **Blargg's tests** - Classic Game Boy test ROMs | ||
| - **Mooneye tests** - GB/GBC accuracy tests | ||
| - **Acid tests** - GPU/graphics tests | ||
| - **Samesuite tests** - Various test scenarios | ||
| - **ax6 tests** - Additional accuracy tests | ||
| - **daid tests** - Custom test collection | ||
| - **hacktix tests** - Edge case tests | ||
| - **cpp tests** - C++ emulator tests | ||
| - **mealybug tests** - Timing and accuracy tests |
There was a problem hiding this comment.
Can links be added to the repos those are pulled from? Also, the descriptions are fairly vague; IMO, either make this an inline list (“The framework includes test ROMs from sources such as Blagg's tests, Mooneye's suite, the Acid Tests, SameSuite...”), or write descriptions that point out highlights of each test ROM set.
There was a problem hiding this comment.
I'd just remove the descriptions, unless those test suites provide their own which we can base on. And remove the repetitive "tests" suffixes. This can be a list of titles which are linked to their pages, just like the emulator list.
There was a problem hiding this comment.
Well, Blargg's suite is generally considered “GB emulator basics”, for example. (Though we may want to present that info on the website as well, more generally.)
There was a problem hiding this comment.
I just meant that if certain test suites are self-described as being about audio, SGB, hblank tricks, etc, that's something we'd want to point out.
But if most of the suites are just "basics"/"various"/"accuracy"/"more accuracy"/"additional various", then descriptions aren't doing much for the reader.
E.g. Blargg's repo just says "Collection of Game Boy test roms"; SameSuite just says "a set of Game Boy and Game Boy Color test ROMs"; etc. Maybe we could distinguish them by pointing out things like "blargg's are from way back in the 2000s" or "SameSuite was written specifically to help develop SameBoy".
| ## Usage | ||
|
|
||
| ### Basic Usage | ||
|
|
||
| Run all tests on all configured emulators: | ||
|
|
||
| ```bash | ||
| python main.py | ||
| ``` | ||
|
|
||
| ### Filter by Emulator | ||
|
|
||
| Test only specific emulators: | ||
|
|
||
| ```bash | ||
| python main.py --emulator mgba --emulator sameboy | ||
| ``` | ||
|
|
||
| ### Filter by Test | ||
|
|
||
| Run only specific test categories: | ||
|
|
||
| ```bash | ||
| python main.py --test blargg --test mooneye | ||
| ``` | ||
|
|
||
| ### Filter by Model | ||
|
|
||
| Test only specific Game Boy models: | ||
|
|
||
| ```bash | ||
| python main.py --model DMG # Original Game Boy | ||
| python main.py --model CGB # Game Boy Color | ||
| python main.py --model SGB # Super Game Boy | ||
| ``` | ||
|
|
||
| ### Get Startup Time Measurements | ||
|
|
||
| ```bash | ||
| python main.py --get-startuptime | ||
| ``` | ||
|
|
||
| ### Get Runtime Measurements | ||
|
|
||
| ```bash | ||
| python main.py --get-runtime | ||
| ``` | ||
|
|
||
| ### Export Test Data | ||
|
|
||
| Export emulator and test definitions to JSON: | ||
|
|
||
| ```bash | ||
| python main.py --dump-emulators-json | ||
| python main.py --dump-tests-json | ||
| ``` |
There was a problem hiding this comment.
This is a good idea in theory, but it's a really verbose way to specify all that. I'd prefer pointing out, in text, that main.py is the test runner, and that one can run python main.py --help to learn about all its knobs.
| ## Project Structure | ||
|
|
||
| ``` | ||
| GBEmulatorShootout/ | ||
| ├── main.py # Main test runner | ||
| ├── build.py # HTML report generator | ||
| ├── emulator.py # Base emulator interface | ||
| ├── test.py # Test framework utilities | ||
| ├── util.py # Helper functions | ||
| ├── requirements.txt # Python dependencies | ||
| ├── emulators/ # Emulator-specific implementations | ||
| │ ├── bdm.py | ||
| │ ├── mgba.py | ||
| │ ├── sameboy.py | ||
| │ └── ... | ||
| └── testroms/ # Test ROM collections | ||
| ├── blargg/ | ||
| ├── mooneye/ | ||
| ├── acid/ | ||
| └── ... | ||
| ``` | ||
|
|
||
| ## How It Works | ||
|
|
||
| 1. **Test Discovery**: The framework discovers all available test ROMs from various test suites | ||
| 2. **Emulator Setup**: Each emulator is configured and prepared for testing | ||
| 3. **Automated Testing**: Tests run automatically using pyautogui for UI automation and screenshot capture | ||
| 4. **Result Analysis**: Screenshots are compared against expected results | ||
| 5. **Report Generation**: Results are compiled into an interactive HTML table | ||
|
|
||
| ## Adding New Emulators | ||
|
|
||
| To add a new emulator: | ||
|
|
||
| 1. Create a new file in `emulators/` implementing the base emulator interface | ||
| 2. Add the emulator specification to `EMULATOR_SPECS` in `main.py` | ||
| 3. Implement required methods: `setup()`, `run()`, `undoSetup()` | ||
|
|
||
| Example emulator specification: | ||
| ```python | ||
| { | ||
| 'factory': lambda: _new_instance("emulators.myemu", "MyEmulator"), | ||
| 'keywords': ["MyEmulator", "myemu"], | ||
| 'name': "MyEmulator", | ||
| 'url': "https://myemulator.example.com/", | ||
| } | ||
| ``` |
There was a problem hiding this comment.
This itself is good information, but I think it would be better moved in a separate CONTRIBUTING.md document for better organisation.
There was a problem hiding this comment.
Agreed. Also there should be a final list item for updating the documentation.
| - 🐛 Report bugs or inaccuracies | ||
| - ✨ Add support for new emulators | ||
| - 📚 Improve documentation | ||
| - 🧪 Add new test ROMs | ||
| - 🎨 Enhance the HTML report |
There was a problem hiding this comment.
This list is in itself very generic, I'd prefer if it linked to appropriate sections of the CONTRIBUTING.md document (mentioned in the previous comment). If the appropriate section doesn't exist, write a new section with a quick summary of how to perform the appropriate contribution kind, and link to the new section from this list.
There was a problem hiding this comment.
That, and the emojis are just noise.
|
|
||
| --- | ||
|
|
||
| **Happy testing!** 🕹️ If you find this project useful, please consider giving it a ⭐ |
There was a problem hiding this comment.
Begging for GitHub stars is not in the spirit of our project, but thank you for the effort 😁
| --- | |
| **Happy testing!** 🕹️ If you find this project useful, please consider giving it a ⭐ |
| @@ -0,0 +1,212 @@ | |||
| # GBEmulatorShootout 🎮 | |||
There was a problem hiding this comment.
| # GBEmulatorShootout 🎮 | |
| # GBEmulatorShootout 🎮🔫 |
Let's have some fun, shall we? :3
There was a problem hiding this comment.
Let's not encourage even more emojis. :3
| # GBEmulatorShootout 🎮 | |
| # GBEmulatorShootout |
| @@ -0,0 +1,212 @@ | |||
| # GBEmulatorShootout 🎮 | |||
There was a problem hiding this comment.
Let's not encourage even more emojis. :3
| # GBEmulatorShootout 🎮 | |
| # GBEmulatorShootout |
|
|
||
| ## Overview | ||
|
|
||
| GBEmulatorShootout automatically tests multiple Game Boy emulators against a suite of accuracy test ROMs, generating detailed HTML reports with pass/fail status and screenshots for each test case. |
There was a problem hiding this comment.
s/statuses/results/ and s/obtained/generated/, but yes.
| ## Supported Emulators | ||
|
|
||
| The framework currently supports testing the following emulators: | ||
|
|
||
| | Emulator | URL | | ||
| |----------|-----| | ||
| | [Beaten Dying Moon](https://mattcurrie.com/bdm-demo/) | mattcurrie.com/bdm-demo | | ||
| | [mGBA](https://mgba.io/) | mgba.io | | ||
| | [KiGB](http://kigb.emuunlim.com/) | kigb.emuunlim.com | | ||
| | [SameBoy](https://sameboy.github.io/) | sameboy.github.io | | ||
| | [bgb](https://bgb.bircd.org/) | bgb.bircd.org | | ||
| | [VisualBoyAdvance](https://sourceforge.net/projects/vba) | sourceforge.net/projects/vba | | ||
| | [VisualBoyAdvance-M](https://github.com/visualboyadvance-m/visualboyadvance-m) | github.com/visualboyadvance-m | | ||
| | [No$gmb](https://problemkaputt.de/gmb.htm) | problemkaputt.de/gmb | | ||
| | [GambatteSpeedrun](https://github.com/pokemon-speedrunning/gambatte-speedrun) | github.com/pokemon-speedrunning/gambatte-speedrun | | ||
| | [Emulicious](https://emulicious.net/) | emulicious.net | | ||
| | [Goomba](https://www.dwedit.org/gba/goombacolor.php) | dwedit.org/gba/goombacolor | | ||
| | [binjgb](https://github.com/binji/binjgb) | github.com/binji/binjgb | | ||
| | [PyBoy](https://github.com/Baekalfen/PyBoy) | github.com/Baekalfen/PyBoy | | ||
| | [ares](https://ares-emu.net/) | ares-emu.net | | ||
| | [Emmy](https://emmy.n1ark.com/) | emmy.n1ark.com | | ||
| | [gameroy](https://github.com/Rodrigodd/gameroy) | github.com/Rodrigodd/gameroy | | ||
| | [DocBoy](https://github.com/Docheinstein/docboy) | github.com/Docheinstein/docboy | | ||
|
|
There was a problem hiding this comment.
That said, it's weird for the table-style "list" to have visible URLs that aren't linked. Linking them would be redundant with the titles, so just remove the visible URLs; anyone can easily hover/click on the titles to see the pages.
| - **Blargg's tests** - Classic Game Boy test ROMs | ||
| - **Mooneye tests** - GB/GBC accuracy tests | ||
| - **Acid tests** - GPU/graphics tests | ||
| - **Samesuite tests** - Various test scenarios | ||
| - **ax6 tests** - Additional accuracy tests | ||
| - **daid tests** - Custom test collection | ||
| - **hacktix tests** - Edge case tests | ||
| - **cpp tests** - C++ emulator tests | ||
| - **mealybug tests** - Timing and accuracy tests |
There was a problem hiding this comment.
I'd just remove the descriptions, unless those test suites provide their own which we can base on. And remove the repetitive "tests" suffixes. This can be a list of titles which are linked to their pages, just like the emulator list.
| ### Basic Usage | ||
|
|
||
| Run all tests on all configured emulators: | ||
|
|
There was a problem hiding this comment.
Sub-sub-headings are excessive for such a short document, and redundant with the brief sentences anyway.
| ### Basic Usage | |
| Run all tests on all configured emulators: | |
| Run all tests on all configured emulators: | |
| ├── mooneye/ | ||
| ├── acid/ | ||
| └── ... | ||
| ``` |
There was a problem hiding this comment.
This can be moved to an ARCHITECTURE.md file like rgbds does.
| ## Project Structure | ||
|
|
||
| ``` | ||
| GBEmulatorShootout/ | ||
| ├── main.py # Main test runner | ||
| ├── build.py # HTML report generator | ||
| ├── emulator.py # Base emulator interface | ||
| ├── test.py # Test framework utilities | ||
| ├── util.py # Helper functions | ||
| ├── requirements.txt # Python dependencies | ||
| ├── emulators/ # Emulator-specific implementations | ||
| │ ├── bdm.py | ||
| │ ├── mgba.py | ||
| │ ├── sameboy.py | ||
| │ └── ... | ||
| └── testroms/ # Test ROM collections | ||
| ├── blargg/ | ||
| ├── mooneye/ | ||
| ├── acid/ | ||
| └── ... | ||
| ``` | ||
|
|
||
| ## How It Works | ||
|
|
||
| 1. **Test Discovery**: The framework discovers all available test ROMs from various test suites | ||
| 2. **Emulator Setup**: Each emulator is configured and prepared for testing | ||
| 3. **Automated Testing**: Tests run automatically using pyautogui for UI automation and screenshot capture | ||
| 4. **Result Analysis**: Screenshots are compared against expected results | ||
| 5. **Report Generation**: Results are compiled into an interactive HTML table | ||
|
|
||
| ## Adding New Emulators | ||
|
|
||
| To add a new emulator: | ||
|
|
||
| 1. Create a new file in `emulators/` implementing the base emulator interface | ||
| 2. Add the emulator specification to `EMULATOR_SPECS` in `main.py` | ||
| 3. Implement required methods: `setup()`, `run()`, `undoSetup()` | ||
|
|
||
| Example emulator specification: | ||
| ```python | ||
| { | ||
| 'factory': lambda: _new_instance("emulators.myemu", "MyEmulator"), | ||
| 'keywords': ["MyEmulator", "myemu"], | ||
| 'name': "MyEmulator", | ||
| 'url': "https://myemulator.example.com/", | ||
| } | ||
| ``` |
There was a problem hiding this comment.
Agreed. Also there should be a final list item for updating the documentation.
| - 🐛 Report bugs or inaccuracies | ||
| - ✨ Add support for new emulators | ||
| - 📚 Improve documentation | ||
| - 🧪 Add new test ROMs | ||
| - 🎨 Enhance the HTML report |
There was a problem hiding this comment.
That, and the emojis are just noise.
| ## License | ||
|
|
||
| This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. | ||
|
|
||
| ## Acknowledgments | ||
|
|
||
| - All the emulator developers for their amazing work | ||
| - The test ROM authors (Blargg, mooneye, and others) | ||
| - The Game Boy development community | ||
|
|
||
| --- | ||
|
|
||
| **Happy testing!** 🕹️ If you find this project useful, please consider giving it a ⭐ |
There was a problem hiding this comment.
The license is clear from the existence of a LICENSE file. The emulators and tests are clearly acknowledged by listing them and linking to them earlier in this document. Overall the tone here is "verbosely friendly customer service voice", not "simple positive but to-the-point README".
| ## License | |
| This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. | |
| ## Acknowledgments | |
| - All the emulator developers for their amazing work | |
| - The test ROM authors (Blargg, mooneye, and others) | |
| - The Game Boy development community | |
| --- | |
| **Happy testing!** 🕹️ If you find this project useful, please consider giving it a ⭐ |
There was a problem hiding this comment.
It remains common to mention the license in the README, if only because sometimes the LICENSE file doesn't apply to the whole repo (there may be exceptions). I think it's fine to keep one line about it.
3 participants
Summary 📝
This PR adds a comprehensive README.md to help new users and contributors understand the project. The repository currently lacks documentation, which makes it difficult for people to get started with the emulator testing framework.
What's included 👍
Closes
Fixes #40
Testing 🧪
Let me know if you'd like any changes! Happy to iterate on this. 😊