BCSV Column Names

[zbanks]

Hey I just wanted to reach out - thanks for sharing what you've discovered so far!

I've been trying to understand how fish spawn, and I wanted to share some progress I made on getting BCSV column names.

I don't have a ton of progress, but I can confirm that the column names are based on CRC32 and I was able to match up some of the details from the names that you gave them (like the months, and nw/n1/...). It looks like the column names must be pretty long on average; and none of the columns are <=5 characters.

(No "issue" here - just figured this is a ~good way to leave a comment.)

[Treeki]

Interesting... I hadn't managed to figure out any names there. Unfortunately, your URL doesn't work so I can't see what's there :x

[zbanks]

Sorry, fixed the link

[Treeki]

Awesome, that gets us a lot closer! The format appears to be as follows: crc32('FieldName FieldType')

So, for example, the columns inside IndoorPhotoStudioItemParam are as follows:

• d5a8b7fe - Unknown
• 20cb67bc - ItemID u16 (this is what I called 'insect_id')
• 374d00da - PosX s16
• 4d8d53ba - PosZ s16
• c61c279a - Unknown
• b5980451 - ReFabric s16
• 7d016b27 - RoomIndex s16
• 54706054 - UniqueID u16

It's worth noting that the columns are sorted alphabetically, so we know for instance that c61c279a lies somewhere between starting with Po and ReF.

I suspect that one sticking point for this is going to be the same issue that affected me when trying to figure out the names in the savefiles (see my additions from today to this repo). While we know the values for enums and strings, we don't know what the type name is, what naming convention they use, or how much of that convention transfers into the source for these hashes.

With these it's even more of a pain than the savefile because the type and the name are both part of the same string, so you've got to guess them both at once...

[Treeki]

I've just pushed a new version that adds ~300 names. There's still quite a few missing, including all of the enums, but it's a start!

The CRC32 trickery in your writeup was incredibly helpful as I was able to reproduce that to figure out the string types - there's one file which uses a couple of common field names (among all of the game's BCSVs) with 30-byte and 60-byte string lengths (when usually they're 32 or 64), and that single bit flip was enough to tip me off to what was going on as I could compare it to the buffer sizes.

[zbanks]

Awesome work!

I was just taking a look at what you originally called the "name" field, 0x87bf00e8. It is nearly identical to 3 other string columns -- it looks like they're all suffixed with their string length, which makes sense with what you just said (maybe you already figured this out?)

• 0x87bf00e8 ends in 32
• 0x13ab5198 ends in 64
• 0x69b161c4 ends in 30
• 0x3febc642 ends in 50

[zbanks]

Also a weird note -- the columns don't seem to be sorted alphabetically in the 1.0.0 version; only in 1.1.1

Compare

# From specs_100.py
class IndoorPhotoStudioItemParam(Row):
	_d5a8bf7e = Enum(0xd5a8bf7e, enum_CompassNESW)
	PosX = S16(0x374d00da)
	RoomIndex = S16(0x7d016b27)
	UniqueID = U16(0x54706054)
	ItemID = U16(0x20cb67bc)
	PosZ = S16(0x4d8d53ba)
	ReBody = S16(0xc61c279a)
	ReFabric = S16(0xb5980451) # size is 4, could this be an array?

# From specs_111.py
class IndoorPhotoStudioItemParam(Row):
	_d5a8bf7e = Enum(0xd5a8bf7e, enum_CompassNESW)
	ItemID = U16(0x20cb67bc)
	PosX = S16(0x374d00da)
	PosZ = S16(0x4d8d53ba)
	ReBody = S16(0xc61c279a)
	ReFabric = S16(0xb5980451)
	RoomIndex = S16(0x7d016b27)
	UniqueID = U16(0x54706054) # size is 4, could this be an array?

Edit: It's funny, InsectAppearParam is mostly chronological in 1.0.0. I wouldn't be surprised if 1.0.0 is the "developer" order and a sort got introduced when exporting the 1.1.1 version.

[Treeki]

Interesting, you're completely right... That said, there's one quirk I didn't realise originally: the fields are sorted first by required alignment and then by name. So first you get all the 32-bit integers/floats, then you get all the 16-bit integers, and then you get all the bytes and strings mixed up together.

I ran hashcat again and managed to crack that one damn string - it's just Label. I may have to revisit the whole list just in case there's anything else I missed >.<