Optimise golbat by jfberry · Pull Request #332 · UnownHash/Golbat

jfberry · 2026-01-26T18:07:47Z

Moved all cache items to pointers
Implemented new locking mechanism (removed striped mutex)
Moved to an alternative pattern for detecting changes - dirty flag in object, since this is a shared object
Changes to fields through Set(.. methods - in line with opaque protobufs which we'll also move to soon
Upgrade to null/v6
Ensure all api objects are returned in dedicated structures
Split decoder files as they're getting pretty big
Add facility to back off some frequent updates (thanks @Fabio1988)
Move webhook generation to structures rather than maps
Move sharding implementation to generic, and add sharding to pokestop + spawnpoint
Move to a pokemon sweeper to update wild pokemon if an encounter doesn't turn up soon to reduce total go-routines

The combination of all of these items gives a reduction to about 40% of previous cpu for my instance

Thanks to Claude or I wouldn't have bothered!

decoder/pokestop.go

Fabio1988 · 2026-01-27T17:33:25Z

I like the approach, even though it makes the files bigger and adding more content, but if that concept works and we can reduce garbage collector work, this definitely has some benefit :)

can you pls rebase? some conflicts on decoder/fort.go :)

jfberry · 2026-01-27T17:34:43Z

I like the approach, even though it makes the files bigger and adding more content, but if that concept works and we can reduce garbage collector work, this definitely has some benefit :)

can you pls rebase? some conflicts on decoder/fort.go :)

If you like the concept, then I should work on applying this to more objects so we can see if we can see the difference

Instead of spawning a goroutine for each wild pokemon that sleeps waiting for an encounter, use a single pending queue with a background sweeper. This reduces memory pressure and goroutine overhead under high load. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Resolved conflicts by keeping typed webhook structs from json_structure while using dirty flag pattern (oldValues, newRecord) for change detection. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…nto dirtyflag_prototype

Copilot

Pull request overview

Explores a “dirty flag + in-place mutation” pattern to reduce allocations/GC pressure by storing pointers in caches and only persisting records when modified, with additional debug instrumentation and a pending Pokémon queue.

Changes:

Refactors several cached DB-backed structs to use dirty flags + setter methods, and updates caches to store pointers (including a new generic sharded cache for concurrency).
Updates webhook generation to compare against captured “old values” instead of passing old/new copies.
Adds a pending Pokémon queue + sweeper to delay wild updates until an encounter arrives (or timeout).

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
routes.go	Introduces a typed `StatusResponse` for consistent JSON responses.
main.go	Initializes the new Pokémon pending queue.
decoder/weather.go	Applies dirty-flag + old-values snapshotting to weather + webhook payload struct.
decoder/tappable.go	Applies dirty-flag setters and pointer caching for tappables.
decoder/stats.go	Updates stats logic to use in-object `oldValues` rather than old/new parameters.
decoder/station.go	Applies dirty-flag setters + old-values snapshotting for station + webhooks.
decoder/spawnpoint.go	Applies dirty-flag setters + pointer caching for spawnpoints.
decoder/sharded_cache.go	Adds a generic sharded cache wrapper around `ttlcache` to reduce lock contention.
decoder/s2cell.go	Switches S2 cell cache values to pointers and updates bulk-save behavior.
decoder/routes.go	Applies dirty-flag setters + pointer caching for routes.
decoder/pokestop_showcase.go	Routes showcase field updates through new Pokestop setters.
decoder/pokestop.go	Major refactor to dirty-flag setters, pointer caching, and webhook comparison via `oldValues`.
decoder/pokemonRtree.go	Updates eviction handling for pointer-based, sharded Pokémon cache.
decoder/player.go	Applies dirty-flag setters + pointer caching for player records.
decoder/pending_pokemon.go	Adds pending Pokémon queue + background sweeper to process expired entries.
decoder/main.go	Switches caches to pointer storage, adds sharded caches, updates cache helpers.
decoder/incident.go	Applies dirty-flag setters + old-values snapshotting for incidents + webhooks.
decoder/gym.go	Major refactor to dirty-flag setters, pointer caching, and webhook comparison via `oldValues`.
decoder/fort.go	Switches fort change webhooks to a typed payload struct; updates shared-field copy logic.
decoder/db_debug.go / decoder/db_debug_off.go	Adds build-tag controlled DB debug logging.
decoder/api_pokemon_scan_v1.go / v2.go / v3.go	Updates API builders for pointer-based Pokémon cache values.
decoder/api_pokemon.go	Updates Pokémon search API to return pointer-based cache values.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

decoder/pokestop.go

decoder/station.go

decoder/gym.go

decoder/fort.go

decoder/pokestop.go

Feat: reduce updates config

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

routes.go:505

GET /api/pokestop/id/:fort_id now uses decoder.PeekPokestopRecord, which is cache-only. If the pokestop isn’t already in cache, this will return null with 202 rather than loading from DB (previous behavior). Either restore a DB-backed getter for this endpoint or return a 404 when pokestop == nil to avoid silently returning empty responses.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

decoder/incident.go

Copilot · 2026-01-31T06:50:56Z

decoder/sharded_cache.go

+func NewShardedCache[K comparable, V any](config ShardedCacheConfig[K, V]) *ShardedCache[K, V] {
+	sc := &ShardedCache[K, V]{
+		shards:     make([]*ttlcache.Cache[K, V], config.NumShards),
+		keyToShard: config.KeyToShard,
+	}
+
+	for i := 0; i < config.NumShards; i++ {
+		opts := []ttlcache.Option[K, V]{
+			ttlcache.WithTTL[K, V](config.TTL),
+		}
+		if config.DisableTouchOnHit {
+			opts = append(opts, ttlcache.WithDisableTouchOnHit[K, V]())
+		}
+		sc.shards[i] = ttlcache.New[K, V](opts...)
+		go sc.shards[i].Start()
+	}
+
+	return sc
+}
+
+// getShard returns the cache shard for the given key
+func (sc *ShardedCache[K, V]) getShard(key K) *ttlcache.Cache[K, V] {
+	return sc.shards[sc.keyToShard(key)%uint64(len(sc.shards))]
+}


NewShardedCache/getShard assume config.NumShards > 0 and config.KeyToShard != nil. If either is zero/nil, getShard will panic (divide by zero / nil function call). Add input validation (and/or sensible defaults) in NewShardedCache and return an error or panic with a clear message.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

routes.go:505

GetPokestop now uses PeekPokestopRecord (cache-only) and returns 202 with a potentially nil body. This changes the endpoint semantics vs DB-backed reads and makes cache misses look like successes. Consider restoring a DB-backed read with a timeout (or returning 404 when the cache miss occurs). Also remove the commented-out context/cancel code.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

decoder/tappable.go

Copilot · 2026-01-31T09:39:58Z

decoder/sharded_cache.go

+// NewShardedCache creates a new sharded cache with the given configuration.
+// The keyToShard function converts keys to uint64 for shard selection.
+func NewShardedCache[K comparable, V any](config ShardedCacheConfig[K, V]) *ShardedCache[K, V] {
+	sc := &ShardedCache[K, V]{
+		shards:     make([]*ttlcache.Cache[K, V], config.NumShards),
+		keyToShard: config.KeyToShard,
+	}
+


NewShardedCache does not validate NumShards (>0) or that KeyToShard is non-nil. As a public constructor, invalid config will panic later (division by zero / nil call). Add upfront validation (return error or panic with a clear message).

Copilot

Pull request overview

Copilot reviewed 63 out of 64 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-31T16:16:24Z

decoder/fort.go

 func CreateFortWebhooks(ctx context.Context, dbDetails db.DbDetails, ids []string, fortType FortType, change FortChange) {
-	var gyms []Gym
-	var stops []Pokestop
 	if fortType == GYM {
 		for _, id := range ids {
-			gym, err := GetGymRecord(ctx, dbDetails, id)
-			if err != nil {
-				continue
-			}
-			if gym == nil {
+			gym, unlock, err := GetGymRecordReadOnly(ctx, dbDetails, id)
+			if err != nil || gym == nil {
+				if unlock != nil {
+					unlock()
+				}
 				continue
 			}
-			gyms = append(gyms, *gym)
+
+			fort := InitWebHookFortFromGym(gym)
+			unlock()
+
+			CreateFortWebHooks(fort, &FortWebhook{}, change)
 		}
 	}
 	if fortType == POKESTOP {
 		for _, id := range ids {
-			stop, err := GetPokestopRecord(ctx, dbDetails, id)
-			if err != nil {
+			stop, unlock, err := getPokestopRecordReadOnly(ctx, dbDetails, id)
+			if err != nil || stop == nil {
+				if unlock != nil {
+					unlock()
+				}
 				continue
 			}
-			if stop == nil {
-				continue
-			}
-			stops = append(stops, *stop)
+
+			fort := InitWebHookFortFromPokestop(stop)
+			unlock()
+
+			CreateFortWebHooks(fort, &FortWebhook{}, change)
 		}
 	}


CreateFortWebhooks always calls CreateFortWebHooks(fort, &FortWebhook{}, change). This is only correct for REMOVAL; for NEW or EDIT it will emit webhooks with an empty new payload (and MatchStatsGeofence will use 0/0 coords), and stats may be attributed to the wrong fort type. Adjust the call site to pass (nil, fort) for NEW, (fort, nil) for REMOVAL, and (oldFort, newFort) for EDIT (or split the helper per change type).

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Webhooks to structures

215fcb2

Fabio1988 reviewed Jan 27, 2026

View reviewed changes

decoder/pokestop.go Outdated Show resolved Hide resolved

jfberry and others added 4 commits January 28, 2026 11:40

Remove comment

22afd74

Example dirty flag implementation

3de2ece

Update pokemon and gym to use new pattern

d238589

jfberry force-pushed the dirtyflag_prototype branch from eb6983d to d238589 Compare January 28, 2026 11:45

jfberry and others added 9 commits January 28, 2026 12:35

Update isNewRecord logic

80e1826

Merge branch 'json_structure' into dirtyflag_prototype

8226a5f

Resolved conflicts by keeping typed webhook structs from json_structure while using dirty flag pattern (oldValues, newRecord) for change detection. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/feature/pokemon-pending-queue' i…

c502d6d

…nto dirtyflag_prototype

Additional types converted to new model

2b5b7fb

Generic sharding introduced for pokestop/gym

a985123

Database change tracing

bb8210c

Add to cache on get

33e6e2e

Add to cache on get incident

7db7d36

Add weather to cache on read

af8b3ef

Fabio1988 requested a review from Copilot January 30, 2026 07:48

Copilot started reviewing on behalf of Fabio1988 January 30, 2026 07:49 View session

Copilot AI reviewed Jan 30, 2026

View reviewed changes

jfberry added 7 commits January 30, 2026 08:39

copilot comments

f89eab3

switch to internal pokemon lock

f4b3714

optimise locking in getOrCreatePokemonRecord

9660a71

optimise locking in getOrCreatePokemonRecord

9e24751

optimise locking in getOrCreatePokemonRecord

ca4f0d6

Pokestop to new locking model

0825445

Gym locking model changes

642eb32

Fabio1988 requested a review from Copilot January 30, 2026 16:51

Copilot started reviewing on behalf of Fabio1988 January 30, 2026 16:51 View session

jfberry added 3 commits January 30, 2026 21:05

Merge pull request #334 from UnownHash/db_update_frequ

0a3cee9

Feat: reduce updates config

more locking update

f4976ac

spawnpoint locking

0f2bd79

Fabio1988 requested a review from Copilot January 31, 2026 06:43

Copilot started reviewing on behalf of Fabio1988 January 31, 2026 06:44 View session

Copilot AI reviewed Jan 31, 2026

View reviewed changes

Fabio1988 requested a review from Copilot January 31, 2026 09:34

Copilot started reviewing on behalf of Fabio1988 January 31, 2026 09:35 View session

Copilot AI reviewed Jan 31, 2026

View reviewed changes

jfberry added 2 commits January 31, 2026 14:22

Tidy source

07efbe6

Remove json fields, these should not be directly serialised

58c22db

jfberry changed the title ~~[For discussion] Example dirty flag implementation~~ Optimise golbat Jan 31, 2026

Claude review items

b69b5ac

This was referenced Jan 31, 2026

Convert webhooks maps to structures #331

Closed

Fix go vet warnings #329

Closed

Replace per-pokemon goroutines with pending queue #328

Closed

Fabio1988 requested review from Fabio1988 and Copilot January 31, 2026 16:10

Copilot started reviewing on behalf of Fabio1988 January 31, 2026 16:11 View session

Copilot AI reviewed Jan 31, 2026

View reviewed changes

jfberry added 4 commits February 1, 2026 17:00

Move config options into tuning

1452c1f

Some fields weren't being updated through SetXX

edf6d76

Improved dbupdate logging to show before/after

140e36a

Ability to turn off nearby cell pokemon

ba42ef9

jfberry mentioned this pull request Feb 1, 2026

Disable Cell Mon #147

Closed

jfberry added 3 commits February 1, 2026 22:37

API documentation

dba646a

Change null debug output so it is more readable

c02ade8

fix location of reduce_updates

0382c1d

Conversation

jfberry commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Fabio1988 commented Jan 27, 2026

Uh oh!

jfberry commented Jan 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jfberry commented Jan 26, 2026 •

edited

Loading