Issue for a little big dataset

[luborliu]

Hi,

I am not sure if this is an issue. Today, I am trying to use the implementation for a bigger dataset, which is actually not really big. It is about 30MB. However, the program is stuck at the map step, do you have any idea for this? I am afraid something is wrong with GraphX API but not quite sure.

Bo

[aizook]

How many points do you have in your dataset ?
The cartesian function that computes the list of edges generates at least n^2 objects in memory whereas of few of them is useful (<eps).
I'm thinking about optimizing this stage to manage much bigger datasets.

[luborliu]

Well, you are right. That is one problem. But actually there's a bigger issue in the following code. The one modification I did in last merge has a big problem. It will generate edges (n^2) then do intersection which will cause the complexity to n^4. I will later update a new version to solve this. Then maybe we can think about how to work on the issue you pointed out.