Explorer crashes when requesting an address with too many transactions

[robvanmieghem]

Even crasges the host os

[LeeSmet]

Investigation shows the host os did not go down, but OOM killed server processes before killing the tfchain daemon. An active ssh connection also hung for about 10 mins before eventually recovering, though logs do not show why this happened

[LeeSmet]

Currently the server handles the load fine (with curl). cpu about 30-40 %. Memory is also more than fine. The response for the call causing the daeomon to be OOM'd returns about 252MB worth of json. Another call currently returning about 520MB worth of json also seems to run fine

[robvanmieghem]

pfew, 520 MB of json for an http call, if it were only the website, we could easily add paging but I'm afraid of the Mobicage wallet.

[GlenDC]

Such huge amounts is only a problem for block creator wallets. No block creator wallet is used by a Mobicage wallet user. And we know for sure it won't be used ever.

Paginating for the explorer could also be done, but I prefer to not spent more effort on the old explorer it until we know for sure we want to keep that alive. For now I am of the opinion we want to migrate to a new explorer as soon as possible. But I guess we can agree on those things by the end of this week, beginning of next week, after we played all well enough and iterated on the new explorer MVP.

[LeeSmet]

The caddy logs show that around the time the explorer crashed a lot of hash lookups came in simultaniously:

- - [28/Oct/2019:11:49:57 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                          [1084/1954]
- - [28/Oct/2019:11:49:57 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:57 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:51:17 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:51:18 +0100] "GET /explorer/hashes/0195de96da59de0bd59c416e96d17df1a5bbc80acb6b02a1db0cde0bcdffca55a4f7f369e955ef HTTP/2.0" 502 16
...

Source IP removed

[robvanmieghem]

Regardless of the new explorer, paging does not seem like a bad idea