Create some subregions for large symbol files

[UsernameFodder]

arm9.yml and overlay29.yml are getting quite large. Now that resymgen (#52) and pmdsky-debug (#57) can handle subregions, splitting some off for the large symbol files could make them less unwieldy. We should decide on a logical way to split these files up. This will also require splitting the function prototype files in headers/functions/ to match the subregion structure in symbols/.

[Frostbyte0x70]

Is this really necessary? Not being able to Ctrl+F in those files seems like a huge con to me.

[UsernameFodder]

I guess it depends on what you're using the symbol files for:

1. If you're just using the generated output tables for Ghidra/No$, this wouldn't change anything.
2. If you're a programmatic consumer of the YAML files (skytemple-files/c-of-time) this makes things a bit more annoying to parse, but it'd be a one-time change.
3. If you're using it as a lookup table for a specific name, then I suppose this would make searching slightly more inconvenient. Even then, this doesn't really seem like a "huge con" to me 🤷. At worst it seems like a slight annoyance. IMO, it's not hard to use tools slightly more powerful than Ctrl+F. E.g., you could use GitHub's repo search feature, or grep/git grep, and I think most IDEs have a way to search all files or a subset of files (VSCode does, which is what I use; and speaking of, I'm pretty sure you can open VSCode in the browser when viewing source files in GitHub).
4. If you're looking through the symbol files for a certain category of symbols (e.g., everything to do with RNG, or memory allocation, or dungeon generation) or treating the symbol files like documentation, having symbols logically grouped into separate files would make it easier to browse. It's the same principle as splitting source code across different files.

The last point especially raises the issue of size. overlay29.yml is already almost 4000 lines long (arm9.yml is close behind), and we're not even close to recording everything, so it'll only get bigger. What happens when it reaches 10k, or 20k? That seems way too big, especially for the documentation purpose. And at some point GitHub will refuse to render the file contents in the source code browser, which will make it hard to link to specific symbols when discussing stuff.

I don't see any downsides to splitting things up besides minor inconvenience, and I think the upsides are pretty big.

[UsernameFodder]

I should clarify, I don't think we necessarily need to do anything right now, since the file sizes are still manageable. For now, at least. I expect things to get out of hand eventually, but we could always put this off until the file sizes actually start causing problems.

I think we'll probably have to do this at some point, though. Maybe after Frostbyte0x70#1 has progressed further.

[Frostbyte0x70]

The last point especially raises the issue of size. overlay29.yml is already almost 4000 lines long (arm9.yml is close behind), and we're not even close to recording everything, so it'll only get bigger. What happens when it reaches 10k, or 20k? That seems way too big, especially for the documentation purpose. And at some point GitHub will refuse to render the file contents in the source code browser, which will make it hard to link to specific symbols when discussing stuff.

You bring up some good points there. I could try setting up VSCode for this, searching across multiple files won't be an issue then.

[UsernameFodder]

If you don't feel like setting up an environment locally, you can even open VSCode through GitHub in your browser. If you're logged in, press . on one of the repo pages (or change the URI from github.com -> github.dev) to view the repo in a browser-based VSCode.

[Frostbyte0x70]

If you don't feel like setting up an environment locally, you can even open VSCode through GitHub in your browser. If you're logged in, press . on one of the repo pages (or change the URI from github.com -> github.dev) to view the repo in a browser-based VSCode.

Huh, I had no idea that existed. Interesting.

[Metalcape]

How should we choose which symbols stay in the parent file and which go into a subregion? For example, should we keep miscellaneous symbols that can't be easily grouped in the parent file? And if some symbols are thematically related but are split into several blocks of addresses, do we:

• keep them in the parent file,
• place them into multiple subregions of the parent file, or
• place them all inside a single subregion including other symbols, then split this subregion again?

[Frostbyte0x70]

For example, should we keep miscellaneous symbols that can't be easily grouped in the parent file?

That sounds like a good idea. If you have a group of symbols that is related, put them in a subregion while keeping misc. stuff or groups with few symbols in the parent file.

And if some symbols are thematically related but are split into several blocks of addresses

You could just make a subregion and put them there anyway. I don't think that the fact that they are on different address blocks matters since each subregion would go into its own file, right?

[UsernameFodder]

For example, should we keep miscellaneous symbols that can't be easily grouped in the parent file?

That sounds like a good idea. If you have a group of symbols that is related, put them in a subregion while keeping misc. stuff or groups with few symbols in the parent file.

Yeah I think we should try to be conservative about splitting out subregions and not go crazy with it, to keep it as simple as possible. So we should only make subregions if it's an obvious group of related symbols that are largely independent from everything else in the binary, and leave the rest in the main file. Ones that immediately come to mind for me are:

• The beginning of arm9 which has a bunch of system-level utilities like the memory allocator
• The end of arm9 which has functions from the C standard library and libgcc
• The dungeon floor generation stuff in overlay29

And if some symbols are thematically related but are split into several blocks of addresses

You could just make a subregion and put them there anyway. I don't think that the fact that they are on different address blocks matters since each subregion would go into its own file, right?

The current YAML files don't use this feature currently, but each YAML file can have multiple separate blocks. A single block is contiguous, but you can glue together multiple blocks to define a piecewise subregion. Ideally we would want to define address ranges somewhat precisely (subject to future adjustments, of course), since the symbol checker will enforce that no symbols in the main file overlap with the subregion.

That being said, if a piecewise subregion would be required, we should be skeptical of creating it in the first place. IMO the most obvious and natural subregions are going to have contiguous address ranges.

[Metalcape]

Sounds good, this is what I was concerned about because I thought that we could end up creating a subregion and then realize there is unrelated stuff in the middle (which can still happen but at least we try to minimize the chance). So in a sense we'll keep most of the symbols in the main files for now, but subregions will prevent the files from becoming too big as more symbols are added.

if a piecewise subregion would be required, we should be skeptical of creating it in the first place. IMO the most obvious and natural subregions are going to have contiguous address ranges.

I agree, it feels like it would make the structure of the file hard to understand, so maybe it's a thing that could be done in the future when there are no "holes" in a block of addresses and we're sure that no more symbols will be added to it