enhancement: Size of different items/classes of items in save files

[Tedster59]

Use case: I'd want to look at bloated geoscape/tactical saves and be able to see what is actually bloating them. e.g. is there a bunch of missions that aren't being cleaned up? LW squad Infiltration data that could be compressed somehow? Abilities firing way more than they need to saving lots of data? knowing what is actually taking up the space is the first step to being able to optimize it.

[rcd47]

I could probably add something to help with this. Do you have any examples of bloated saves you're trying to analyze? That would help me figure out what kind of UI would be useful.

One well-known cause of bloat is that units and some other stuff don't get cleaned up when transitioning back to strategy. The linked issue specifically calls out units and items. I also noticed recently that XCGS_BattleData doesn't get cleaned up either (though the associated XCGS_MissionSite does).

[Tedster59]

Thanks.

here are some example saves I got from DerAva of later game geoscape saves that are getting bigger

geoscape saves

save_lw140.zip

tactical save:
save_AUTOSAVE- Campaign 306, Mission 161, Turn 24.zip

[rcd47]

I'm eventually going to add a UI to help with this. I don't know when I'll have time to do that, but I spent some time looking at DerAva's save files, and that led me to a major discovery. The problem isn't directly part of his saves, but the aftermath is visible, and prompted me to take a look at my own saves. Luckily, in my last campaign, I was running a mod (currently unreleased) that dumps the history before every map transition. So I have the entire history of that campaign and I was able to find a place where the problem occurs. It's a bug in the base game, and you'll need to make a highlander change to fix it. Here's a history file that I'll refer to below: 432.zip

The problem is loot. Which explains why DerAva has such a bad case, because we all know how much Greedy Commander likes his loot. 😁

Specifically, the problem is loot that awards resources. In the history file I provided, take a look at frame 17138. This frame is awarding supplies and alloys obtained from clearing a room on the Avenger. The first thing to notice is that the loot system is poorly optimized to handle large quantities of resources. Clearing that room awarded 50 supplies, but instead of creating one XCGS_Item that has Quantity = 50, the game created 50 XCGS_Items, each representing one supply.

That's not great, but it would be fine as long as the items get properly deleted. Except they don't. Take a look at frame 17141. The 50 loot items get merged into the XCGS_Item that represents supplies in the player's inventory (object 587) but the 50 loot items don't get removed. So they just sit there for the rest of the campaign. This problem is amplified by the fact that pretty much everything you could get as loot is a resource - supplies, alloys, elerium cores, corpses, etc. Even ability points are treated as a resource.

[Tedster59]

oh wow. so we need the equivalent of Tactical State Cleaner but for loot now?

Tactical State Cleaner never got fully integrated into CHL and left as standalone, hopefully a standalone works here too.

[rcd47]

As I understand it, TSC never got integrated into CHL because there's no way to reliably guarantee that the stuff it deletes is actually dead. But after looking at the XCGS_HQXCom code, I think this loot issue could be fixed correctly. PutItemInInventory() already has code that's merging the loot item into the inventory item's quantity. It just doesn't delete the loot item afterwards.

TSC might already be accidentally solving this problem. I haven't looked at the code, but I assume it deletes items that it thinks are unused. Which likely includes all of these resource items.

[rcd47]

Correction: the loot items that get created are almost entirely garbage. The first item (object 7040 in the file I attached) has Quantity = 50. The other 49 items have Quantity = 0. Not sure where the code is that creates these objects but that's definitely a bug too.

Also, I looked at the TSC code. It might have some impact on this issue, but wouldn't fix it fully. It only removes items that are in the inventory of units that it deletes.

[Tedster59]

I just did some basic testing that filters to resources cleanup only via console command, from the test save LW128 that Derava gave me, post mission without the cleanup the size was 3944 KB, with the cleanup it was 3725 KB, so a bit over 200KB saved from that. I did notice some other lingering things such as a PsiZombie_MeleeAttack weapon and probably some other things as well since I was only checking for resources explicitly. Will do more thorough testing

[rcd47]

So ~5% reduction. It might be more substantial than it looks though, because the save file's compression would be very effective in this situation.

I should stop being lazy and dust off my Discord account. Which Discord servers are you active on?

[Tedster59]

Ok, with a more aggressive check, I got near 500KB reduction

Code I used cached a list of all items in HQ inventory, unit inventories, or pending reward states, and would delete anything it didn't find in said cache

static function CacheAllInUseItems(out array<int> ItemsInUse)
{
	local StateObjectReference ItemRef;
	local XComGameState_Unit UnitState;
	local XComGameState_Reward RewardState;

	// Grab everything in HQ inventory
	forEach `XCOMHQ.Inventory (ItemRef)
	{
		ItemsInUse.AddItem(ItemRef.ObjectID);
	}

	// Grab items in rewards
	foreach `XCOMHISTORY.IterateByClassType(class'XComGameState_Reward',RewardState)
	{
		if(RewardState.RewardObjectReference.ObjectID > 0)
		{
			ItemsInUse.AddItem(RewardState.RewardObjectReference.ObjectID);
		}
	}

	// Items on units
	foreach `XCOMHISTORY.IterateByClassType(class'XComGameState_Unit', UnitState)
	{
		foreach UnitState.InventoryItems (ItemRef)
		{
			if(ItemsInUse.Find(ItemRef.ObjectID) == INDEX_NONE)
				ItemsInUse.AddItem(ItemRef.ObjectID);
		}
	}

}

The save does load, but I did not do any further validation to determine if any items are missing that shouldn't be.

[Tedster59]

The main discords I'm on are the LWOTC one and the main X2 modding discord.

https://discord.gg/vv5yJF9

https://discord.gg/ZytsZ93

[Tedster59]

While this is definitely a more substantial improvement, both compressed saves from immediately after a mission are much smaller than the existing save, so I want to look into what is causing the bloat pre-compression so I can either optimize stuff that is being done during Strategy, or maybe come up with a way to compress the game before saving it (a la the Fix Save Bloat experimental command)

[rcd47]

Bloat analysis UI was added by #18 and released in 1.2.0.