Memory leak

[olehz]

20:20

22:15

[michelbio]

Maybe related to mapbox/mapbox-gl-native#15922 ?

[brendan-ward]

@olehz thanks for reporting and for investigating the potential underlying issue from mapbox-gl-native.

Can you tell us more about your usage here to help us investigate?

• what version of mbgl-renderer?
• what OS / container environment?
• what is the load you are throwing at mbgl-renderer - is it sporadic image render requests, are you generating many images in a pipeline (e.g., rendering tiles), etc?

Am I seeing correctly that you have several node processes of mbgl-renderer listening on the same port?

Are you able to reproduce this on NodeJS 10?

[olehz]

There are many requests for generating images in the pipeline. The data source is quite large GeoJSON. I am not able to test this on NodeJS 10.

[ifzm]

I also found a similar problem. As the number of renders increases and the style is a larger geojson, the memory usage will gradually increase without being released.
I don’t know if it’s an upstream problem.

use docker for consbio/mbgl-renderer

[brendan-ward]

Looks like there are indeed memory leaks coming from mapbox-gl-native. Here's a bit of valgrind profiling of the NodeJS 10 version:

==315== 5,310 bytes in 253 blocks are definitely lost in loss record 5,914 of 5,936
==315==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==315==    by 0x4C3AE4A: strdup (strdup.c:42)
==315==    by 0x92EB4A6: ???
==315==    by 0x92EB254: ???
==315==    by 0x92907CC: ???
==315==    by 0x9209436: ??? (in /usr/lib/x86_64-linux-gnu/libGLdispatch.so.0.0.0)
==315==    by 0x9209CE8: __glDispatchMakeCurrent (in /usr/lib/x86_64-linux-gnu/libGLdispatch.so.0.0.0)
==315==    by 0x91877F6: ??? (in /usr/lib/x86_64-linux-gnu/libEGL.so.1.1.0)
==315==    by 0xB56DC9D: mbgl::gl::EGLBackendImpl::activateContext() (headless_backend_egl.cpp:124)
==315==    by 0xB421CE4: activate (backend_scope.cpp:56)
==315==    by 0xB421CE4: mbgl::gfx::BackendScope::BackendScope(mbgl::gfx::RendererBackend&, mbgl::gfx::BackendScope::ScopeType) (backend_scope.cpp:31)
==315==    by 0xB3CE8CC: operator() (headless_frontend.cpp:29)
==315==    by 0xB3CE8CC: std::_Function_handler<void (), mbgl::HeadlessFrontend::HeadlessFrontend(mbgl::Size, float, mbgl::gfx::ContextMode, std::experimental::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >)::$_0>::_M_invoke(std::_Any_data const&) (functional:1871)
==315==    by 0xA711E0: uv__async_io (async.c:147)
==315== 
==315== 12,448 (368 direct, 12,080 indirect) bytes in 1 blocks are definitely lost in loss record 5,925 of 5,936
==315==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==315==    by 0x15E67C9: icu_64::DecimalFormat::clone() const (in /usr/local/bin/node)
==315==    by 0x15DEC76: icu_64::NumberFormat::createInstance(icu_64::Locale const&, UErrorCode&) (in /usr/local/bin/node)
==315==    by 0x104FBDC: v8::internal::(anonymous namespace)::CreateICUNumberFormat(v8::internal::Isolate*, icu_64::Locale const&, v8::internal::Handle<v8::internal::JSObject>) (in /usr/local/bin/node)
==315==    by 0x1051094: v8::internal::NumberFormat::InitializeNumberFormat(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::JSObject>) (in /usr/local/bin/node)
==315==    by 0x11796D6: v8::internal::Runtime_CreateNumberFormat(int, v8::internal::Object**, v8::internal::Isolate*) (in /usr/local/bin/node)
==315==    by 0x120F6065C1D7: ???
==315==    by 0x120F6068FA14: ???
==315==    by 0x120F606118D4: ???
==315==    by 0x120F606118D4: ???
==315==    by 0x120F606118D4: ???
==315==    by 0x120F6060A5C2: ???
==315== 
==315== 40,032 (32 direct, 40,000 indirect) bytes in 1 blocks are definitely lost in loss record 5,933 of 5,936
==315==    at 0x4838DEF: operator new(unsigned long) (vg_replace_malloc.c:342)
==315==    by 0x91C254: node::Buffer::New(v8::Isolate*, char*, unsigned long, void (*)(char*, void*), void*) (in /usr/local/bin/node)
==315==    by 0xB2A9D36: NewBuffer (nan.h:890)
==315==    by 0xB2A9D36: node_mbgl::NodeMap::renderFinished() (node_map.cpp:542)
==315==    by 0xA711E0: uv__async_io (async.c:147)
==315==    by 0xA83B07: uv__io_poll (linux-core.c:431)
==315==    by 0xA71B6A: uv_run (core.c:375)
==315==    by 0x905664: node::Start(v8::Isolate*, node::IsolateData*, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::string, std::allocator<std::string> > const&) (in /usr/local/bin/node)
==315==    by 0x90374E: node::Start(int, char**) (in /usr/local/bin/node)
==315==    by 0x4BD3D09: (below main) (libc-start.c:308)
==315== 
==315== LEAK SUMMARY:
==315==    definitely lost: 9,230 bytes in 329 blocks
==315==    indirectly lost: 53,281 bytes in 54 blocks
==315==      possibly lost: 3,528 bytes in 16 blocks

Will need to repeat this with the maplibre-gl-native bindings once those are more widely available.

Related brendan-ward/pymgl#1

[brendan-ward]

Possibly related: mapbox-gl-native #15915

It looks like some of the Node-related memory leaks were solved in 15922 but not actually merged; these may still need to be addressed in the latest maplibre-gl-native NodeJS bindings.