Crash under high memory pressure

[pljakobs]

I can pretty reliably drive ConfigDB to crash when an ESP8266 is relatively low on free Heap (usually due to a relatively large number of http requests - like loading the web page) and then fetching one of the larger database streams (/config returns a ~3.5kB object, /data a 6kB object)
it seems to happen more frequently on the /config endpoint, despite the correspondign object being smaller, but app-config, the db behind it, consists of multiple stores.

181711629 [CFGDB] open 'app-config/color.json' failed
181727068 [CFGDB] open 'app-config/sync.json' failed
E:M 104
E:M 96
E:M 96
E:M 448
E:M 448
E:M 448

Does splitting a structure into multiple stores increase the memory footprint significantly, when streaming the whole database?

I am gating my api endpoints by checking for what I think should be enough free heap, for the /info and /config endpoints, I check for 12kB free and otherwise send a "TOO_MANY_REQUESTS" and ask the client to back off for as second, but 12kB seems not to be enough for my config structure.

stacktrace.txt

[pljakobs]

I have just re-tested with the whole /config structure in a single store and got a pretty similar crash.

94963658 [CFGDB] open 'app-config/color.json' failed
94980800 [CFGDB] open 'app-config/sync.json' failed
95056234 onInfo
E:M 616
E:M 616
E:M 624
E:M 616
E:M 632
E:M 624
E:M 616
E:M 616
E:M 624
E:M 616
E:M 624
E:M 624
E:M 616
E:M 616
E:M 624
E:M 616
E:M 632
E:M 624
E:M 616
E:M 616
E:M 624
E:M 616
E:M 616
E:M 624
E:M 616
E:M 616
E:M 624
E:M 616
E:M 632
E:M 624
E:M 616
E:M 616
E:M 632
E:M 616
E:M 616
E:M 624
E:M 624
E:M 616
E:M 624
E:M 624
E:M 616
E:M 616
E:M 632
E:M 616
E:M 616
E:M 624
E:M 616
E:M 616
E:M 616
E:M 616
E:M 616
E:M 616
E:M 616
E:M 632
E:M 632
E:M 616
E:M 616
E:M 624
E:M 632
E:M 624
E:M 616
E:M 616
E:M 632
E:M 624
E:M 616
E:M 616
E:M 616
E:M 624
E:M 624
E:M 616
E:M 616
E:M 616
E:M 616
E:M 624
E:M 632
E:M 616
E:M 616
E:M 616
E:M 624
E:M 624
95145792 Hostname lightinator-10964518 not in list adding to hostname db
E:M 208
E:M 448
E:M 448
E:M 448


***** Fatal exception 29 (STORE_PROHIBITED)
pc=0x402039f7 sp=0x3ffffba0 excvaddr=0x00000000
ps=0x00000030 sar=0x00000004 vpri=0x700ffc00
r00: 0x40203b55=1075854165 r01: 0x3ffffba0=1073740704 r02: 0x00000000=         0 
r03: 0x00000004=         4 r04: 0x00000000=         0 r05: 0x00000002=         2 
r06: 0x60000010=1610612752 r07: 0x00000000=         0 r08: 0x00000000=         0 
r09: 0x00000000=         0 r10: 0x3ffef560=1073673568 r11: 0x00000000=         0 
r12: 0x00000000=         0 r13: 0x00000001=         1 r14: 0x3ffffd74=1073741172 
r15: 0x3ffe93c4=1073648580 

stacktrace.txt

[pljakobs]

ha! I had - for no good reason, I guess - configured the httpServer with maxActiveConnections=40 - not sure if that was me or what I was thinking, but turning this down to 10 not only makes the server more responsive but also more resilient.

Not sure though, if this just create an impossible environment for ConfigDB or if there is something that can be done to make it a bit more resilient in low memory situations.

[pljakobs]

it seems that with settings.maxActiveConnections = 10; the issue gets less frequent but isn't entirely gone.
If I interpret what is happening correctly, then this happens when after the freeHeap gate has been passed another request comes in and eats into the available memory.
I guess it's hard to do anything about that.

[mikee47]

There may be places in ConfigDB which lack checking on returned pointers. To evaluate this is probably easiest on the host emulator with debugging enabled and malloc_count so we can see all the allocations going on. The Basic_Config sample is built with it, and you can enable logging like this:

MallocCount::enableLogging(true);

// Default is to log all allocations of 256 bytes or more, can be adjusted easily
MallocCount::setLogThreshold(1024);