Support parsing HyperlocalExperiment #276

jfberry · 2025-07-15T10:34:45Z

Interested in others thoughts about moving updated to a ms field (where others are not). Consistency might be more important than other concerns (I'd be willing to consider a PR moving to a virtual updated and every other table being updated to ms as the per second resolution annoys me too)

I think it is fine to just do this instead to keep compatibility. I imagine this would break compatibility for a lot of other things if we introduce this to older tables though?

You would use a virtual column (as a calculated field) to maintain the old name. Anyway, for the time being we should change this to updated

Why? This is a new table and RM PR is already implemented for this. Do you mean we should add a virtual updated column for this instead?

Also since the column explicitly gives the unit (which is actually a convenient consequence of following the same naming convention as proto), I think it's fine to just keep this as is.

jfberry · 2025-07-15T10:36:56Z

Urgh. I understand there may be no unique identifier (unusual for niantic though this is!) but I wonder if lat/lon as a float works reliably given the round-trip to database (I doubt it will be guaranteed byte-exact). We may be forced for the key to move to an integer representing a fixed precision - but I haven't seen the resolution of the data that comes from niantic for this - how many decimal places is it?

I think it should be exact. From my data gathered three months ago, these fields are always truncated to 1e-6 (ignoring floating point rounding), e.g. 20.681438,-156.442944.

Full row: 3578,1743375172270,1743376072270,20.681438,-156.442944,100.00000000000001,spawn_wild_bug_types,1743376070600

I guess if testing turns out to be problematic, we can store (int)(1e6*lat) instead. (It didn't seem problematic in the data I collected.)

jfberry · 2025-07-15T10:37:51Z

this is not considering lat/lon - there is a method for nearby precision comparison of floats
updated wouldn't normally be considered here I don't think

They aren't considered because they are primary keys. :)

jfberry · 2025-07-15T10:40:05Z

I wonder if the hash would be better as a method on the key - ... I think the stripedmutex just calls the standard hashing interface from memory

stripedmutex takes a string key so this is more performant.

jfberry · 2025-07-15T10:40:21Z

why? equality is generally best avoided with floats (though 0 generally works as is a special float)

These are unlikely to be a result of a floating point math calculation so I think precise comparison is fine. Also the old code is comparing with 0 as well instead of something like > 1e-12.

And yes 0.0 is a special float in proto, indicating default value.

jfberry · 2025-07-15T10:44:36Z

We don't normally force engine (some people use alternates), charset or collate (forcing and not using database default gives problems on a join)
Suggest that you probably need an index on lat,lon,end_ms for the standard map type query
and don't need ix_updated_ms as this won't be used anywhere
the end_ms is likely used to expire

I think I copied this ENGINE part from a similar sql migration script.

Just following existing codebase.
Updated keys/indices as suggested.

I will PR the removal of these, this is a mistake

-Original file line number
+Diff line change
@@ Expand Up / @@ -50,6 +50,7 @@ type cleanup struct { @@
     	Quests      bool `koanf:"quests"`
     	Incidents   bool `koanf:"incidents"`
     	Tappables   bool `koanf:"tappables"`
+    	Hyperlocals bool `koanf:"hyperlocals"`
     	Stats       bool `koanf:"stats"`
     	StatsDays   int  `koanf:"stats_days"`
     	DeviceHours int  `koanf:"device_hours"`
@@ Expand Down Expand Up / @@ -124,18 +125,19 @@ type tuning struct { @@
     }
     type scanRule struct {
-    	Areas            []string       `koanf:"areas"`
-    	AreaNames        []geo.AreaName `koanf:"-"`
-    	ScanContext      []string       `koanf:"context"`
-    	ProcessPokemon   *bool          `koanf:"pokemon"`
-    	ProcessWilds     *bool          `koanf:"wild_pokemon"`
-    	ProcessNearby    *bool          `koanf:"nearby_pokemon"`
-    	ProcessWeather   *bool          `koanf:"weather"`
-    	ProcessCells     *bool          `koanf:"cells"`
-    	ProcessPokestops *bool          `koanf:"pokestops"`
-    	ProcessGyms      *bool          `koanf:"gyms"`
-    	ProcessStations  *bool          `koanf:"stations"`
-    	ProcessTappables *bool          `koanf:"tappables"`
+    	Areas              []string       `koanf:"areas"`
+    	AreaNames          []geo.AreaName `koanf:"-"`
+    	ScanContext        []string       `koanf:"context"`
+    	ProcessPokemon     *bool          `koanf:"pokemon"`
+    	ProcessWilds       *bool          `koanf:"wild_pokemon"`
+    	ProcessNearby      *bool          `koanf:"nearby_pokemon"`
+    	ProcessWeather     *bool          `koanf:"weather"`
+    	ProcessCells       *bool          `koanf:"cells"`
+    	ProcessPokestops   *bool          `koanf:"pokestops"`
+    	ProcessGyms        *bool          `koanf:"gyms"`
+    	ProcessStations    *bool          `koanf:"stations"`
+    	ProcessTappables   *bool          `koanf:"tappables"`
+    	ProcessHyperlocals *bool          `koanf:"hyperlocals"`
     }
     var Config configDefinition

-Original file line number
+Diff line change
@@ Expand Up / @@ -44,6 +44,7 @@ func ReadConfig() (configDefinition, error) { @@
     			Quests:      true,
     			Incidents:   true,
     			Tappables:   true,
+    			Hyperlocals: true,
     			StatsDays:   7,
     			DeviceHours: 24,
     		},
@@ Expand Down @@

-Original file line number
+Diff line change
@@ -0,0 +1,127 @@
+    package decoder
+    import (
+    	"context"
+    	"database/sql"
+    	"errors"
+    	"golbat/db"
+    	"golbat/pogo"
+    	"github.com/jellydator/ttlcache/v3"
+    	log "github.com/sirupsen/logrus"
+    )
+    // Hyperlocal struct for hyperlocal experiment data
+    type Hyperlocal struct {
+    	ExperimentId      int32   `db:"experiment_id" json:"experiment_id"`
+    	StartMs           int64   `db:"start_ms" json:"start_ms"`
+    	EndMs             int64   `db:"end_ms" json:"end_ms"`
+    	Lat               float64 `db:"lat" json:"lat"`
+    	Lon               float64 `db:"lon" json:"lon"`
+    	RadiusM           float64 `db:"radius_m" json:"radius_m"`
+    	ChallengeBonusKey string  `db:"challenge_bonus_key" json:"challenge_bonus_key"`
+    	UpdatedMs         int64   `db:"updated_ms" json:"updated_ms"`
+    }
+    // HyperlocalKey represents the composite primary key for hyperlocal records
+    type HyperlocalKey struct {
+    	ExperimentId int32   `json:"experiment_id"`
+    	Lat          float64 `json:"lat"`
+    	Lon          float64 `json:"lon"`
+    }
+    func (h *Hyperlocal) getKey() HyperlocalKey {
+    	return HyperlocalKey{
+    		ExperimentId: h.ExperimentId,
+    		Lat:          h.Lat,
+    		Lon:          h.Lon,
+    	}
+    }
+    func (h *Hyperlocal) updateFromHyperlocalProto(data *pogo.HyperlocalExperimentClientProto, timestampMs int64) {
+    	h.ExperimentId = data.ExperimentId
+    	h.StartMs = data.StartMs
+    	h.EndMs = data.EndMs
+    	h.Lat = data.LatDegrees
+    	h.Lon = data.LngDegrees
+    	h.RadiusM = data.EventRadiusM
+    	h.ChallengeBonusKey = data.ChallengeBonusKey
+    	h.UpdatedMs = timestampMs
+    }
+    func getHyperlocalRecord(ctx context.Context, db db.DbDetails, key HyperlocalKey) (*Hyperlocal, error) {
+    	// Check cache first using HyperlocalKey directly
+    	if cachedItem := hyperlocalCache.Get(key); cachedItem != nil {
+    		hyperlocal := cachedItem.Value()
+    		return &hyperlocal, nil
+    	}
+    	hyperlocal := Hyperlocal{}
+    	err := db.GeneralDb.GetContext(ctx, &hyperlocal,
+    		`SELECT experiment_id, start_ms, end_ms, lat, lon, radius_m, challenge_bonus_key, updated_ms
+             FROM hyperlocal
+             WHERE experiment_id = ? AND lat = ? AND lon = ?`, key.ExperimentId, key.Lat, key.Lon)
+    	statsCollector.IncDbQuery("select hyperlocal", err)
+    	if errors.Is(err, sql.ErrNoRows) {
+    		return nil, nil
+    	}
+    	if err != nil {
+    		return nil, err
+    	}
+    	return &hyperlocal, nil
+    }
+    func saveHyperlocalRecord(ctx context.Context, details db.DbDetails, hyperlocal *Hyperlocal) {
+    	key := hyperlocal.getKey()
+    	oldHyperlocal, _ := getHyperlocalRecord(ctx, details, key)
+    	if oldHyperlocal != nil && !hasChangesHyperlocal(oldHyperlocal, hyperlocal) {
+    		return
+    	}
+    	if oldHyperlocal == nil {
+    		res, err := details.GeneralDb.NamedExecContext(ctx, `
+    			INSERT INTO hyperlocal (
+    				experiment_id, start_ms, end_ms, lat, lon, radius_m, challenge_bonus_key, updated_ms
+    			) VALUES (
+    				:experiment_id, :start_ms, :end_ms, :lat, :lon, :radius_m, :challenge_bonus_key, :updated_ms
+    			)
+    			`, hyperlocal)
+    		statsCollector.IncDbQuery("insert hyperlocal", err)
+    		if err != nil {
+    			log.Errorf("insert hyperlocal %+v: %s", key, err)
+    			return
+    		}
+    		_ = res
+    	} else {
+    		res, err := details.GeneralDb.NamedExecContext(ctx, `
+    			UPDATE hyperlocal SET
+    				start_ms = :start_ms,
+    				end_ms = :end_ms,
+    				radius_m = :radius_m,
+    				challenge_bonus_key = :challenge_bonus_key,
+    				updated_ms = :updated_ms
+    			WHERE experiment_id = :experiment_id AND lat = :lat AND lon = :lon
+    			`, hyperlocal)
+    		statsCollector.IncDbQuery("update hyperlocal", err)
+    		if err != nil {
+    			log.Errorf("update hyperlocal %+v: %s", key, err)
+    			return
+    		}
+    		_ = res
+    	}
+    	hyperlocalCache.Set(key, *hyperlocal, ttlcache.DefaultTTL)
+    }
+    func hasChangesHyperlocal(old *Hyperlocal, new *Hyperlocal) bool {
+    	return old.StartMs != new.StartMs ||
+    		old.EndMs != new.EndMs ||
+    		old.RadiusM != new.RadiusM ||
+    		old.ChallengeBonusKey != new.ChallengeBonusKey ||
+    		old.UpdatedMs < new.UpdatedMs
+    }
+    func ClearHyperlocalCache() {
+    	hyperlocalCache.DeleteAll()
+    }

-Original file line number
+Diff line change
@@ Expand Up / @@ -53,6 +53,11 @@ type RawMapPokemonData struct { @@
     	Timestamp int64
     }
+    type RawHyperlocalData struct {
+    	Data      *pogo.HyperlocalExperimentClientProto
+    	Timestamp int64
+    }
     type webhooksSenderInterface interface {
     	AddMessage(whType webhooks.WebhookType, message any, areas []geo.AreaName)
     }
@@ Expand All / @@ -72,6 +77,7 @@ var playerCache *ttlcache.Cache[string, Player] @@
     var routeCache *ttlcache.Cache[string, Route]
     var diskEncounterCache *ttlcache.Cache[uint64, *pogo.DiskEncounterOutProto]
     var getMapFortsCache *ttlcache.Cache[string, *pogo.GetMapFortsOutProto_FortProto]
+    var hyperlocalCache *ttlcache.Cache[HyperlocalKey, Hyperlocal]
     var gymStripedMutex = stripedmutex.New(128)
     var pokestopStripedMutex = stripedmutex.New(128)
@@ Expand All / @@ -82,6 +88,7 @@ var pokemonStripedMutex = intstripedmutex.New(1024) @@
     var weatherStripedMutex = intstripedmutex.New(128)
     var s2cellStripedMutex = stripedmutex.New(1024)
     var routeStripedMutex = stripedmutex.New(128)
+    var hyperlocalStripedMutex = intstripedmutex.New(128)
     var s2CellLookup = sync.Map{}
@@ Expand Down Expand Up / @@ -168,6 +175,11 @@ func initDataCache() { @@
     		ttlcache.WithTTL[string, Route](60 * time.Minute),
     	)
     	go routeCache.Start()
+    	hyperlocalCache = ttlcache.New[HyperlocalKey, Hyperlocal](
+    		ttlcache.WithTTL[HyperlocalKey, Hyperlocal](60 * time.Minute),
+    	)
+    	go hyperlocalCache.Start()
     }
     func InitialiseOhbem() {
@@ Expand Down Expand Up @@
     	}
     }
+    func UpdateHyperlocalBatch(ctx context.Context, db db.DbDetails, scanParameters ScanParameters, p []RawHyperlocalData) {
+    	if len(p) <= 0 {
+    		return
+    	}
+    	for _, raw := range p {
+    		key := HyperlocalKey{
+    			ExperimentId: raw.Data.GetExperimentId(),
+    			Lat:          raw.Data.GetLatDegrees(),
+    			Lon:          raw.Data.GetLngDegrees(),
+    		}
+    		hyperlocalMutex, _ := hyperlocalStripedMutex.GetLock(uint64(key.ExperimentId) ^ math.Float64bits(key.Lat) ^ math.Float64bits(key.Lon))
+    		hyperlocalMutex.Lock()
+    		hyperlocal, err := getHyperlocalRecord(ctx, db, key)
+    		if err != nil {
+    			log.Errorf("getHyperlocalRecord: %s", err)
+    			hyperlocalMutex.Unlock()
+    			continue
+    		}
+    		if hyperlocal == nil {
+    			hyperlocal = &Hyperlocal{}
+    		}
+    		hyperlocal.updateFromHyperlocalProto(raw.Data, raw.Timestamp)
+    		saveHyperlocalRecord(ctx, db, hyperlocal)
+    		hyperlocalMutex.Unlock()
+    	}
+    }
     func UpdatePokemonBatch(ctx context.Context, db db.DbDetails, scanParameters ScanParameters, wildPokemonList []RawWildPokemonData, nearbyPokemonList []RawNearbyPokemonData, mapPokemonList []RawMapPokemonData, weather []*pogo.ClientWeatherProto, username string) {
     	weatherLookup := make(map[int64]pogo.GameplayWeatherProto_WeatherCondition)
     	for _, weatherProto := range weather {
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up @@
     			} else {
     				infoData["pokemon_id"] = int(info.GetPokemonId())
     			}
-    			if info.ShinyProbability > 0.0 {
+    			if info.ShinyProbability != 0.0 {
     				infoData["shiny_probability"] = info.ShinyProbability
     			}
     			if display := info.PokemonDisplay; display != nil {
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -7,15 +7,16 @@ import ( @@
     )
     type ScanParameters struct {
-    	ProcessPokemon   bool
-    	ProcessWild      bool
-    	ProcessNearby    bool
-    	ProcessWeather   bool
-    	ProcessPokestops bool
-    	ProcessGyms      bool
-    	ProcessStations  bool
-    	ProcessCells     bool
-    	ProcessTappables bool
+    	ProcessPokemon     bool
+    	ProcessWild        bool
+    	ProcessNearby      bool
+    	ProcessWeather     bool
+    	ProcessPokestops   bool
+    	ProcessGyms        bool
+    	ProcessStations    bool
+    	ProcessHyperlocals bool
+    	ProcessCells       bool
+    	ProcessTappables   bool
     }
     func FindScanConfiguration(scanContext string, lat, lon float64) ScanParameters {
@@ Expand Down Expand Up @@
     			return *value
     		}
     		return ScanParameters{
-    			ProcessPokemon:   defaultTrue(rule.ProcessPokemon),
-    			ProcessWild:      defaultTrue(rule.ProcessWilds),
-    			ProcessNearby:    defaultTrue(rule.ProcessNearby),
-    			ProcessCells:     defaultTrue(rule.ProcessCells),
-    			ProcessWeather:   defaultTrue(rule.ProcessWeather),
-    			ProcessPokestops: defaultTrue(rule.ProcessPokestops),
-    			ProcessGyms:      defaultTrue(rule.ProcessGyms),
-    			ProcessStations:  defaultTrue(rule.ProcessStations),
-    			ProcessTappables: defaultTrue(rule.ProcessTappables),
+    			ProcessPokemon:     defaultTrue(rule.ProcessPokemon),
+    			ProcessWild:        defaultTrue(rule.ProcessWilds),
+    			ProcessNearby:      defaultTrue(rule.ProcessNearby),
+    			ProcessCells:       defaultTrue(rule.ProcessCells),
+    			ProcessWeather:     defaultTrue(rule.ProcessWeather),
+    			ProcessPokestops:   defaultTrue(rule.ProcessPokestops),
+    			ProcessGyms:        defaultTrue(rule.ProcessGyms),
+    			ProcessStations:    defaultTrue(rule.ProcessStations),
+    			ProcessTappables:   defaultTrue(rule.ProcessTappables),
+    			ProcessHyperlocals: defaultTrue(rule.ProcessHyperlocals),
     		}
     	}
     	return ScanParameters{
-    		ProcessPokemon:   true,
-    		ProcessWild:      true,
-    		ProcessNearby:    true,
-    		ProcessCells:     true,
-    		ProcessWeather:   true,
-    		ProcessGyms:      true,
-    		ProcessPokestops: true,
-    		ProcessStations:  true,
-    		ProcessTappables: true,
+    		ProcessPokemon:     true,
+    		ProcessWild:        true,
+    		ProcessNearby:      true,
+    		ProcessCells:       true,
+    		ProcessWeather:     true,
+    		ProcessGyms:        true,
+    		ProcessPokestops:   true,
+    		ProcessStations:    true,
+    		ProcessTappables:   true,
+    		ProcessHyperlocals: true,
     	}
     }

-Original file line number
+Diff line change
@@ Expand Up / @@ -226,6 +226,10 @@ func main() { @@
     		StartTappableExpiry(db)
     	}
+    	if cfg.Cleanup.Hyperlocals == true {
+    		StartHyperlocalExpiry(db)
+    	}
     	if cfg.Cleanup.Quests == true {
     		StartQuestExpiry(db)
     	}
@@ Expand Down Expand Up @@
     	var newMapPokemon []decoder.RawMapPokemonData
     	var newMapCells []uint64
     	var cellsToBeCleaned []uint64
+    	var newHyperlocals []decoder.RawHyperlocalData
     	for _, mapCell := range decodedGmo.MapCell {
     		if isCellNotEmpty(mapCell) {
@@ Expand All @@
     		for _, station := range mapCell.Stations {
     			newStations = append(newStations, decoder.RawStationData{Cell: mapCell.S2CellId, Data: station})
     		}
+    		for _, hyperlocal := range mapCell.HyperlocalExperiment {
+    			newHyperlocals = append(newHyperlocals, decoder.RawHyperlocalData{Data: hyperlocal, Timestamp: mapCell.AsOfTimeMs})
+    		}
     	}
     	if scanParameters.ProcessGyms || scanParameters.ProcessPokestops {
@@ Expand All @@
     	if scanParameters.ProcessStations {
     		decoder.UpdateStationBatch(ctx, dbDetails, scanParameters, newStations)
     	}
+    	if scanParameters.ProcessHyperlocals {
+    		decoder.UpdateHyperlocalBatch(ctx, dbDetails, scanParameters, newHyperlocals)
+    	}
     	if scanParameters.ProcessCells {
     		decoder.UpdateClientMapS2CellBatch(ctx, dbDetails, newMapCells)
@@ Expand Down @@

-Original file line number
+Diff line change
@@ -0,0 +1,14 @@
+    CREATE TABLE `hyperlocal` (
+     `experiment_id`       INT NOT NULL,
+     `start_ms`            BIGINT NOT NULL,
+     `end_ms`              BIGINT NOT NULL,
+     `lat`                 DOUBLE(18,14) NOT NULL,
+     `lon`                 DOUBLE(18,14) NOT NULL,
+     `radius_m`            DOUBLE(18,14) NOT NULL,
+     `challenge_bonus_key` VARCHAR(255) NOT NULL,
+     `updated_ms`          BIGINT NOT NULL,
+     PRIMARY KEY(`experiment_id`,`lat`,`lon`),
+     KEY `ix_end_ms` (`end_ms`,`lat`,`lon`)
+    ) ENGINE = InnoDB
+      DEFAULT CHARSET = utf8mb4
+      COLLATE = utf8mb4_general_ci;

Support parsing HyperlocalExperiment #276

Are you sure you want to change the base?

Uh oh!

Support parsing HyperlocalExperiment #276

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mygod Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Mygod Jul 15, 2025 •

edited

Loading