Addition of live stats api endpoint

[Petap0w]

Adds an api endpoint to get count of active pokemon, active with IV, active 100iv and active shiny.

Call with /api/pokemon/livestats and response like {"pokemon_active":18853,"pokemon_active_iv":13077,"pokemon_active_100iv":5,"pokemon_active_shiny":56}

[jfberry]

I like the idea of this, but it’s kind of expensive because it is going to count every single pokemon in a loop when you call it.

another (almost free) approach would be to keep a running total, which could do this per area (since that is already calculated for every new Pokemon). I guess double counting might be a challenge on Pokemon re-encounter though

[Petap0w]

At the moment the only way to get this info is to use db write of Pokémon table which is way more expensive than calling this endpoint even with hundreds of thousands of Pokémon.
I’ll have a look at what you suggest but my zero knowledge of Golang is slowing me down a bit 🙃

[jfberry]

Agreed (or call the scan with a world range and some filters which is equivalent to this but even more expensive)

[jfberry]

If the live count is straight forward (we do know it’s a re encounter because of other stats data) and removing a pokemon gets a callback also… so it might be ok… then it could be worth it

[Petap0w]

Just for context, with 2 million active pokemon that takes 250ms:
apiLiveStats - 2133307 pokemon_active, 1594005 pokemon_active_iv, 450 pokemon_active_100iv, 6062 pokemon_active_shiny, total time 254.594317ms

[jfberry]

During which time there will be contention to all other activities

[jfberry]

Ah, actually the problem is that you are using the wrong cache.
PokemonLookupCache is intended for fast lookup for scanning api; the cache of all pokemon is pokemonCache -- this is the one that contains the full record. You would not needed to have added expire timestamp or shiny if you were iterating pokemonCache

[Petap0w]

Follow up and testing on cache entries I added some debug log lines to get PokemonCache and PokemonLookupCache counts for total, no timer and expired Pokémon and I confirm expired are not cleaned properly (25% of my cache is expired…)
Expiration is tested with int64(valueOrMinus1(pokemon.ExpireTimestamp)) < now && int64(valueOrMinus1(pokemon.ExpireTimestamp)) > -1

apiLiveStats - PokemonCache : 4134016 pokemon_cached, 204 pokemon_no_timer, 1171489 pokemon_expired, 2960306 pokemon_active, 2238920 pokemon_active_iv, 732 pokemon_active_100iv, 8455 pokemon_active_shiny, total time 6.100892071s
apiLiveStats - PokemonLookupCache : 4153473 pokemon_lookup_cached, 0 pokemon_lookup_no_timer, 1166184 pokemon_lookup_expired, 2985274 pokemon_lookup_active, 2257956 pokemon_lookup_active_iv, 736 pokemon_lookup_active_100iv, total time 6.100938236s

[jfberry]

When did you run this? The expiry looks like it will run the first time after one hour of operation and then continuously afterwards — can you repeat after a while? There may be a way of kickstarting the first run (the library is built to wait an hour when there are no entries to expire it seems)

[Petap0w]

Ran after 8 hours roughly