Search takes too long (options on indexing)

[bbkgh]

Hi. My repo has many files and searching on it takes too time. Is there any option to build index and use that for more performant searchs?

probe search --timeout 60 --max-results 20 '"ViewsTests"' /repo -> Search operation timed out after 60 seconds

[buger]

Hi! What is the size of repository, in terms of files and lines of code? And which language it is?

Also which hardwared and OS do you use?

Thanks!

[Jandev]

I'm having the same issue with the Semantic Kernel repository (~40MB on disk).

By changing the timeout to 90 seconds, I was able to retrieve results. Working on an NVME SSD disk, if it matters.

As for hardware & OS:

OS Name	Microsoft Windows 11 Pro
Version	10.0.26100 Build 26100
Processor	11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz, 3600 Mhz, 8 Core(s), 16 Logical Processor(s)
Installed Physical Memory (RAM)	32,0 GB

[buger]

Thanks! That’s very helpful! Which query I can use to replicate it?

[Jandev]

I've used this one (as I'm using your docs-mcp solution)

C:\Users\jandevries\AppData\Local\npm-cache\_npx\6836e08c2900d844\node_modules\@buger\probe\bin\probe.exe search --max-tokens 10000 --timeout 600 "yaml workflow agent multi-agent user input" "C:\Users\jandevries\AppData\Local\npm-cache\_npx\6836e08c2900d844\node_modules\@buger\docs-mcp\data"

[buger]

@Jandev I just improved performance like 4x or similar. Would love if you will be able to test this change with latest probe 0.6.0 RC

[Jandev]

@Jandev I just improved performance like 4x or similar. Would love if you will be able to test this change with latest probe 0.6.0 RC

Sure thing.
Testing has to wait a bit, as I'm on my Summer break for the upcoming weeks.

[bartlomiejwolk]

I have the same problem with a very large codebase when using the latest probe. Is there a way to reduce execution time? Ripgrep by itself gives results in a couple of seconds.

[buger]

@bartlomiejwolk is it an oss repo, so I can use it as part of my test pack? If not, can you give me info about amount of files, lines of code, and language used?

Probe already use ripgrep internally and it indeed the fastest step. But it does way more, as it have to unwrap every found line, parse ast for it, do the bm25 ranking, and re-rank results for best match.

[bartlomiejwolk]

Unfortunately its work related. It has around 400k files (8k of it are C++ source). Most of the files are text-based assets.

I'm using Probe as MCP. Would be nice that the "maxResults" option influenced the time it takes to produce the output. Currently even when I set it to 10 results it still timeouts.

[buger]

Right, or maybe it search though those assets too now somehow. Can you clarify file extensions of those assets?

[bartlomiejwolk]

those are mainly .yaml files. I need the LLM to search through them also.

[buger]

I had 2 day hackaton on performance, and now it is like 30x faster then on v0.5

@Jandev Now smth like this takes just 2 secondsnpx -y @buger/probe@latest search "yaml workflow agent multi-agent user input" /Users/leonidbugaev/go/src/semantic-kernel --max-tokens 10000!

I'm still validating if it all correct, but so far looks positive!

[buger]

@bbkgh @bartlomiejwolk @Jandev had any chance to compare new performance? Thanks!

[Jandev]

@bbkgh @bartlomiejwolk @Jandev had any chance to compare new performance? Thanks!

I'm still on my summer break. Should be back home next week and able to test/validate.