Large data sets cause browser to crash

[jwildfire]

Summary

Very large data (~100 mb+) sets cause severe performance issues, and may not render at all.

Details

After the data is loaded, the codebook shouldn't need huge amounts of time to summarize the data and render the page. After doing some basic profiling, I'm 99% sure issues are due to inefficient data handling in various places in the code.

[samussiah]

• makeSummary takes ~60% of load time
  • determineType takes a significant amount of that time. Avoid checking every variable value; rather loop through values until a value identifies the variable's type.
  • coerce to numeric once here, and only here
  • define value arrays once
  • figure out how to preserve record index, used when clicking bars; defining an array of objects that is the same size as the input data array is computationally intensive
• draw takes ~30% of load time
  • makeTitle takes a lot of time
  • lots of garbage collection at the bottom of makeHist
  • avoid creating new data arrays in the charts
  • a

[jwildfire]

• makeSummary takes ~60% of load time

  • determineType takes a significant amount of that time. Avoid checking every variable value; rather loop through values until a value identifies the variable's type.

Could also recommend the user provide type for each column and avoid this altogether in large data sets. We could just pass in the R column types in datadigest.

[samussiah]

• makeSummary takes ~60% of load time

  • determineType takes a significant amount of that time. Avoid checking every variable value; rather loop through values until a value identifies the variable's type.

Could also recommend the user provide type for each column and avoid this altogether in large data sets. We could just pass in the R column types in datadigest.

Having R determine the data types would definitely take some load off the browser.