Optimize indexing to handle new data better. #6
Description
As of right now, the indexing is pretty fast, but there's a lot of redundancy with re-running the indexing as new stuff comes in. Not a big deal for smaller sets but lags like crazy when we're dealing with millions of lines.
We should figure out a more clever heuristic to make this more seamless.
Low hanging fruit is to debounce the typing, which I'm kind of doing (well a buffer and polling).
Not sure how much parallel processing is getting us. Might be able to spawn a separate task to do background processing as new things come in.