Refactor raw decoding by comstud · Pull Request #168 · UnownHash/Golbat

comstud · 2023-10-21T09:00:02Z

move device location tracking to its own package/struct
create a generic proto decoder
- use a map of method -> decoder function/required account level
- decoder functions take the full proto info, including scan metadata
clean up the Raw HTTP route
- move Raw gin handler to new http_handler package
- move the proto decoding/processing to http_raw_decoder.
- also allow json array of protos for batching multiple locations
use new grpc_raw_decoder in the grpc server

comstud · 2023-10-21T09:01:02Z

Untested yet, so marked as draft. Early feedback is welcome, though.

comstud · 2023-10-21T09:07:47Z

raw_decoder/raw_decoder.go

+	}
+}
+
+func (dec *rawDecoder) Decode(ctx context.Context, protoData *ProtoData, decodeFn func(context.Context, *Proto)) {


I intend to move the rest of the raw decoding out of 'main' and into this package, but for now this just takes a pointer to main's 'decode' function, shared between the http and grpc apis. This MR already does enough. I'll save more code shuffling for a follow-up.

comstud · 2023-10-21T20:08:57Z

I am running a version of this that is simply rebased on top of the the 'stats collector' PR that I've also made.

If any of this is approved, I would suggest we take a look at the stats collector PR first and land that. Then I can push up the rebased version of this.

And I have the next commit ready that moves the rest of decoding out of main and into the raw_decoder. But it is rebased on top of the rebased version of this... So I'm not going to PR that yet. But if interested in seeing where this is going, this is the commit: comstud@11866f0

jfberry · 2023-10-26T09:17:55Z

raw_decoder/raw_decoder.go

+)
+
+type RawDecoder interface {
+	GetProtoDataFromHTTP(http.Header, []byte) (*ProtoData, error)


Do alternative ways of receiving data make sense to put in the same interface?
Surely we just get the data into a standard for (ProtoData I guess) and then pass on to a common processor?

Yeah, this doesn't feel right to me, either. Not sure what I was smoking that night. Starting to refresh myself with what I did here.. since this was 1.5mo ago. :) I've been running with this patch this whole time, though. I'll fix this.

jfberry · 2023-10-26T09:19:14Z

raw_decoder/raw_decoder.go

+}
+
+func (pd ProtoData) Lat() float64 {
+	l := len(pd.Protos)


I've not followed exactly when this is called - for the standard MITM send, the lat/lon at the top of the body indicates where the calls were made from (and where processing rules are applied against); individual lat/lon on items aren't really used. I know this is what pogodroid does, but this is the exception I think.

I also believe PD to be the exception, but since this object is the general object being built by the /raw decoder, it sort of needs to support it. Although, I don't remember how this is being used either. I will look it over.

jfberry · 2023-10-26T09:23:02Z

raw_decoder/raw_decoder.go

+	for _, proto := range protoData.Protos {
+		// provide independent cancellation contexts for each proto decode
+		ctx, cancel := context.WithTimeout(ctx, dec.decodeTimeout)
+		decodeFn(ctx, &proto)


We used to call each decoder in their own go-routine; however this caused too much tripping up and waiting on the striped mutex - since you might have responses that affect the same objects (eg during questing etc).
You also see some MITM include the gmo and the encounters in the same raw (even though this can't be how they were executed against the game).
Having independent cancellation is fine; originally intention was to be quite strict to not end in a backlog situation (which RDM is very common for when overloaded) - where you never catch up because you have so many concurrent decodes happening.

This is actually copy/paste from existing code. ~~But It looks like we don't need to make a new context for every single call here.~~

(I'll reply to the other comments when I can )

jfberry · 2023-10-26T09:25:53Z

I'm in favour of this refactor (and breaking out the decoders) - the split between the decode module and main has always been uncomfortable - it gradually evolved from golbat the prototype into golbat the production code.
A rethink is very worthwhile

comstud · 2026-01-11T22:16:50Z

I've rebased and re-done this all... after letting this sit for over 2yr. :)

There is more cleanup that can be done since MAD is dead now, but I left it for now.

* move device location tracking to its own package/struct * create a generic proto decoder * use a map of method -> decoder function/required account level * decoder functions take the full proto info, including scan metadata * clean up the Raw HTTP route * move Raw gin handler to new http_handler package * move the proto decoding/processing to http_raw_decoder. * also allow json array of protos for batching multiple locations * use new grpc_raw_decoder in the grpc server

comstud · 2026-01-11T23:56:29Z

I need to give this some good testing, yet.

jfberry · 2026-01-12T15:09:33Z

Please see the change I made to the json decoding for raw in the raw_lessgc branch (never merged). If you are testing anyway this should be easy to bring across and should reduce gc pressure

comstud · 2026-01-12T15:37:39Z

Please see the change I made to the json decoding for raw in the raw_lessgc branch (never merged). If you are testing anyway this should be easy to bring across and should reduce gc pressure

Will do

jfberry · 2026-01-12T17:19:33Z

There is more cleanup that can be done since MAD is dead now, but I left it for now.

Although it's not much code (and segregated), deleting it is fine by me

comstud marked this pull request as draft October 21, 2023 09:00

comstud commented Oct 21, 2023

View reviewed changes

comstud force-pushed the raw-decoding branch from 34710f3 to 2a4b37a Compare October 21, 2023 09:15

jfberry reviewed Oct 26, 2023

View reviewed changes

comstud force-pushed the raw-decoding branch from 2a4b37a to 8807961 Compare December 10, 2023 07:51

comstud force-pushed the raw-decoding branch from 8807961 to b6bab4a Compare January 11, 2026 22:10

comstud changed the title ~~Refactor the initial raw decoding~~ Refactor raw decoding Jan 11, 2026

comstud force-pushed the raw-decoding branch from b6bab4a to 79c7a6f Compare January 11, 2026 22:38

Conversation

comstud commented Oct 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comstud commented Oct 21, 2023

Uh oh!

comstud Oct 21, 2023

Choose a reason for hiding this comment

Uh oh!

comstud commented Oct 21, 2023

Uh oh!

jfberry Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

comstud Dec 10, 2023

Choose a reason for hiding this comment

Uh oh!

jfberry Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

comstud Dec 10, 2023

Choose a reason for hiding this comment

Uh oh!

jfberry Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

comstud Oct 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comstud Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

jfberry commented Oct 26, 2023

Uh oh!

comstud commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comstud commented Jan 11, 2026

Uh oh!

jfberry commented Jan 12, 2026

Uh oh!

comstud commented Jan 12, 2026

Uh oh!

jfberry commented Jan 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

comstud commented Oct 21, 2023 •

edited

Loading

comstud Oct 26, 2023 •

edited

Loading

comstud commented Jan 11, 2026 •

edited

Loading