Socket limit preventing full exports from working with large amounts of entities

[phalestrivir]

Frodo Library version

v3.3.1

Describe the issue

We have a tenant with over 100 IDM config entities, and when we try to export the entities, either using frodo idm export -AD . host or frodo config export -AD . host Frodo terminates, and not only terminates but doesn't terminate with any sort of error indicating what happened.

I already debugged the issue and it appears to be caused by this line in the BaseApi.ts file that sets the max sockets to be 100. Since we are using Promise.all in the exportConfigEntities function in IdmConfigOps.ts, it opens a new socket per each request, and since there are over 100 of them Frodo just terminates without any reason.

As for the solution to the issue, a simple fix would be to just increase the limit, which I confirmed does work. By default the ProxyAgent allows for an infinite amount of sockets, so using the default would be the simplest fix for this issue if we don't want to worry about hitting the limit again. The other option could be to change anywhere in Frodo where we are using Promise.all (or Promise.allSettled) to use a custom promiseAll function that we create that automatically batches (or possibly pools the connections for) the requests so we never exceed the socket limit.

I'm open to other suggestions as well, especially if you have any @hfranklin or @vscheuber. I think upgrading the limit would be a good temporary fix, and then later we can refactor Frodo to batch requests or something so we don't run into issues using more sockets than the specified limit.

[vscheuber]

@phalestrivir nice RCA, man! The setting was set to a value that seemed reasonable at the time. I think a quick fix is the right thing to do for now to allow Frodo to handle larger datasets. In the long run, we might want to look into the more sophisticated approach you suggested but really only if users report issues. If there are issues with unlimited sockets, then those might occur in CI/CD pipeline envs with minimal resources (like GitHub action runners). What do you think, @hfranklin ?

[vscheuber]

So, I was working on an internal and external report of unexplainable failures of frodo config export commands. Turns out they are all caused by this issue you also identified, @phalestrivir .

I am now prioritizing this as I have customers and peers asking for a fix for this.

I've done some research and it appears that leaving it unlimited would be a bad idea as it could drain the host of resources and lead to instability. So I am looking for a better default value that is significantly higher than 100 and should address this issue for the majority of envs which are seeing it today. I am thinking 500 or 1000.

It appears that the default max for many linux distributions is 1024 (ulimit -n) so I am leaning towards 500 as a start. We can then debate if we need to pursue a more reliable fix. @hfranklin

[vscheuber]

Oh boy... this rabbit hole goes deep. Good read and many offered solutions for a permanent fix.

[hfranklin]

I'm inclined to try a pool/queue approach first rather than batching. It should give us better performance. It will probably be pretty clean if we handle it inside the promise function. I'll see about putting together an example of what I am thinking.