Grip crashes with OOM error

[jarifibrahim]

Hi, I'm trying to reproduce the issue filed here dgraph-io/badger#1058 (comment) . I'm running the commands mentioned on the comment a 32 GB RAM machine but GRIP crashes with OOM error.
Here's what the heap profile looks before OOM crash

Fetching profile over HTTP from http://localhost:8080/debug/pprof/heap
Saved profile in /home/ibrahim/pprof/pprof.grip.alloc_objects.alloc_space.inuse_objects.inuse_space.004.pb.gz
File: grip
Build ID: db26589919a25557fa7d0e00f5be9abd198900b6
Type: inuse_space
Time: Nov 22, 2019 at 3:40pm (IST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 11.08GB, 98.98% of 11.19GB total
Dropped 25 nodes (cum <= 0.06GB)
Showing top 10 nodes out of 28
      flat  flat%   sum%        cum   cum%
    5.60GB 50.02% 50.02%     5.60GB 50.02%  reflect.mapassign
    2.70GB 24.15% 74.17%     9.98GB 89.13%  github.com/golang/protobuf/jsonpb.(*Unmarshaler).unmarshalValue
    0.76GB  6.79% 80.96%     0.76GB  6.79%  github.com/bmeg/golib.ReadLines.func1
    0.75GB  6.73% 87.69%     0.86GB  7.69%  reflect.cvtBytesString
    0.41GB  3.69% 91.38%     0.41GB  3.69%  encoding/json.(*RawMessage).UnmarshalJSON
    0.24GB  2.12% 93.51%     0.24GB  2.12%  encoding/json.(*Decoder).refill
    0.21GB  1.87% 95.38%     3.13GB 28.00%  encoding/json.(*decodeState).object
    0.19GB  1.73% 97.10%     0.19GB  1.73%  reflect.Zero
    0.11GB  0.97% 98.07%     0.11GB  0.97%  reflect.New
    0.10GB  0.91% 98.98%     0.10GB  0.91%  github.com/golang/protobuf/jsonpb.unquote
(pprof) top -cum
Showing nodes accounting for 8717.24MB, 76.05% of 11463.14MB total
Dropped 25 nodes (cum <= 57.32MB)
Showing top 10 nodes out of 28
      flat  flat%   sum%        cum   cum%
         0     0%     0% 10568.13MB 92.19%  github.com/bmeg/grip/util.StreamVerticesFromFile.func1
         0     0%     0% 10568.13MB 92.19%  github.com/golang/protobuf/jsonpb.(*Unmarshaler).Unmarshal
         0     0%     0% 10568.13MB 92.19%  github.com/golang/protobuf/jsonpb.(*Unmarshaler).UnmarshalNext
         0     0%     0% 10568.13MB 92.19%  golang.org/x/sync/errgroup.(*Group).Go.func1
 2768.35MB 24.15% 24.15% 10217.65MB 89.13%  github.com/golang/protobuf/jsonpb.(*Unmarshaler).unmarshalValue
         0     0% 24.15%  5734.38MB 50.02%  reflect.Value.SetMapIndex
 5734.38MB 50.02% 74.17%  5734.38MB 50.02%  reflect.mapassign
  214.51MB  1.87% 76.05%  3210.01MB 28.00%  encoding/json.(*decodeState).object
         0     0% 76.05%  3210.01MB 28.00%  encoding/json.(*decodeState).unmarshal
         0     0% 76.05%  3210.01MB 28.00%  encoding/json.(*decodeState).value

(pprof) list github.com/bmeg/grip/util.StreamVerticesFromFile.func1
Total: 11.19GB
ROUTINE ======================== github.com/bmeg/grip/util.StreamVerticesFromFile.func1 in /home/ibrahim/Projects/grip/util/file_reader.go
         0    10.32GB (flat, cum) 92.19% of Total
         .          .     41:
         .          .     42:	for i := 0; i < runtime.NumCPU(); i++ {
         .          .     43:		g.Go(func() error {
         .          .     44:			for line := range reader {
         .          .     45:				v := &gripql.Vertex{}
         .    10.32GB     46:				err := m.Unmarshal(bytes.NewReader(line), v)
         .          .     47:				if err == io.EOF {
         .          .     48:					break
         .          .     49:				}
         .          .     50:				if err != nil {
         .          .     51:					log.WithFields(log.Fields{"error": err}).Errorf("Unmarshaling vertex: %v", line)
(pprof)  

[adamstruck]

I fixed the bug on our side and have tracked the issue back to badger. The snippet (https://gist.github.com/adamstruck/22f94e7a49ebe4fe0550e81a788327eb) I posted with my other issue reproduces the bug. The creation of the byte slices is intentional to simulate unique keys and values being set in badger. These large entries are accumulated within the badger transaction and thus lead to the OOM. I think it may have to do with the calculation in https://github.com/dgraph-io/badger/blob/3eb4e723d3ae4cdde80434ba8cf57b1d19c3fb38/structs.go#L154 that used to determine the overall size of the transaction.

[jarifibrahim]

The creation of the byte slices is intentional to simulate unique keys and values being set in badger.

@adamstruck with this change dgraph-io/badger#1058 (comment) the OOM did not happen.

Badger will go out of memory if you use write batch and keep inserting data with huge values infinitely but I don't think it is related to this issue.
The memory profile I've pasted above doesn't even show badger.

I think it may have to do with the calculation in https://github.com/dgraph-io/badger/blob/3eb4e723d3ae4cdde80434ba8cf57b1d19c3fb38/structs.go#L154 that used to determine the overall size of the transaction.

I don't remember the reason right now but the calculation is done intentionally that way.

         .          .     44:			for line := range reader {
         .          .     45:				v := &gripql.Vertex{}
         .    10.32GB     46:				err := m.Unmarshal(bytes.NewReader(line), v)
         .          .     47:				if err == io.EOF {

Do you think we can optimize the above call in any way?