JavaScript heap out of memory

[DerKorb]

I get a heap out of memory error. I already tried to increase it to 25Gb, but it still does not work. I generated the dataset with entwine and it has around ~50GB in ~150k files. Is that just to much, is there some way I can control conversion so it wont run out of memory?

node --max-old-space-size=25096 /usr/local/bin/ept tile -i ./3d_tiles_entwine/ept.json -o tiled -t 1 --z-offset 25 --force 
Tiling: 3d_tiles_entwine -> tiled
Threads: 1
Z offset: 25
Overwriting output
(node:1940642) V8: /usr/local/share/.config/yarn/global/node_modules/ept-tools/lib/lib/laz-perf.asm.js:2994 Invalid asm.js: Expected numeric literal.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:1940642) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy

<--- Last few GCs --->

[1940642:0x5a1aca0]   421250 ms: Scavenge 16060.4 (16362.8) -> 16059.2 (16362.8) MB, 333.2 / 0.0 ms  (average mu = 0.936, current mu = 0.936) allocation failure; 
[1940642:0x5a1aca0]   421689 ms: Scavenge 16060.4 (16362.8) -> 16060.1 (16362.8) MB, 438.4 / 0.0 ms  (average mu = 0.936, current mu = 0.936) allocation failure; 
[1940642:0x5a1aca0]   422133 ms: Scavenge 16060.4 (16362.8) -> 16060.3 (16362.8) MB, 443.8 / 0.0 ms  (average mu = 0.936, current mu = 0.936) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Scavenger: semi-space copy Allocation failed - JavaScript heap out of memory
 1: 0xb6e500 node::Abort() [ept-tools]
 2: 0xa7e632  [ept-tools]
 3: 0xd47f20 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [ept-tools]
 4: 0xd482c7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [ept-tools]
 5: 0xf25685  [ept-tools]
 6: 0xfaf0be v8::internal::SlotCallbackResult v8::internal::Scavenger::ScavengeObject<v8::internal::FullHeapObjectSlot>(v8::internal::FullHeapObjectSlot, v8::internal::HeapObject) [ept-tools]
 7: 0xfb2198  [ept-tools]
 8: 0xfb3354 v8::internal::Scavenger::Process(v8::JobDelegate*) [ept-tools]
 9: 0xfb7aef v8::internal::ScavengerCollector::JobTask::ProcessItems(v8::JobDelegate*, v8::internal::Scavenger*) [ept-tools]
10: 0xfb7c20 v8::internal::ScavengerCollector::JobTask::Run(v8::JobDelegate*) [ept-tools]
11: 0x1ac6fd4 v8::platform::DefaultJobWorker::Run() [ept-tools]
12: 0xbd7315  [ept-tools]
13: 0x7fd5b99d944b  [/usr/lib/libc.so.6]
14: 0x7fd5b9a5ce40  [/usr/lib/libc.so.6]
^C^C^C^C[1]    1940642 IOT instruction (core dumped)  node --max-old-space-size=25096 /usr/local/bin/ept tile -i  -o tiled -t 1  25

[blemmer]

I have the same issue. Did you find a way to solve it?

[DerKorb]

I am using potree converter and gocesiumtiler by now. Both have their own problems thoughm gocesiumtile can't handle datasets above a certain size and potree is awesome, but its a own format that does not work with cesium.

[lymperis-e]

I had the same issue, and fixed it by adjusting the max heap size of node, using the --max-old-space-size flag when starting the node process.

On windows, I did it by first location the source of ept

Get-Command -Name "ept"

which returned C:\Program Files\nodejs\ept.ps1. I then went on to give node an ample 24Gb of heap, by modifying the script( C:\Program Files\nodejs\ept.ps1) as follows:

#!/usr/bin/env pwsh

$basedir=Split-Path $MyInvocation.MyCommand.Definition -Parent

$exe=""
if ($PSVersionTable.PSVersion -lt "6.0" -or $IsWindows) {
  # Fix case when both the Windows and Linux builds of Node
  # are installed in the same directory
  $exe=".exe"
}
$ret=0
if (Test-Path "$basedir/node$exe") {
  # Support pipeline input
  if ($MyInvocation.ExpectingInput) {
    $input | & "$basedir/node$exe" --max-old-space-size=24576 "$basedir/node_modules/ept-tools/lib/app/index.js" $args
  } else {
    & "$basedir/node$exe" --max-old-space-size=24576 "$basedir/node_modules/ept-tools/lib/app/index.js" $args
  }
  $ret=$LASTEXITCODE
} else {
  # Support pipeline input
  if ($MyInvocation.ExpectingInput) {
    $input | & "node$exe" --max-old-space-size=24576 "$basedir/node_modules/ept-tools/lib/app/index.js" $args
  } else {
    & "node$exe" --max-old-space-size=24576 "$basedir/node_modules/ept-tools/lib/app/index.js" $args
  }
  $ret=$LASTEXITCODE
}
exit $ret

Note1: that 24Gb is maybe a bit of an overkill, I just was lucky to have lots of RAM, so I used to it just to be safe. I monitored the child processes, and they collectively used about 2Gb at the worst point, while tiling a ~5Gb EPT. So adjust this as you please!

Note2: it would be useful to allow for arguments to be passed to node when invoking ept, or include a special argument specifically for the max-heap (?)