Whohandler searches "nlweb_sites" - how to set up?

[oidebrett]

Is there any documentation on how to set up the prerequisites sites for "nlweb_sites" for the WhoHandler?

The code in code/python/core/whoHandler.py in the function whoRetrieveInt tries to search using site='nlweb_sites'

but NLWeb returns "No answers were found for your query." when try to search over all sites

I have followed the guide to load data like this:

python -m data_loading.db_load https://feeds.libsyn.com/121695/rss Behind-the-Tech

The logs show:
=== NLWebHandler INIT === [MULTI-SITE-HANDLER] Initializing with query: what does kevin think of mcp? [MULTI-SITE-HANDLER] Params: {'score': 100, 'justification': 'Only available tool for this query type'} [Using who_endpoint: http://localhost:8000/who Calling sites_from_who_streaming with endpoint=http://localhost:8000/who, query=what does kevin think of mcp? [SSEStreamWrapper] broadcast_message called: conv_id=1763231581610, my_conv_id=1763231581610 [SSEStreamWrapper] Writing SSE data: data: {"message_type": "intermediate_message", "status": "Identifying relevant sites ...", "timestam... [SSEStreamWrapper] SSE data written successfully === NLWebHandler INIT === Query params: {'query': 'what does kevin think of mcp?', 'streaming': 'true'} [MULTI-SITE] WARNING: No sites returned from who endpoint!

Am I correct in assuming that there must be an entry in the vector db that contains a list of sites and this is called nlweb_sites? If so this isnt explained anywhere in the documentation. I am happy to contribute with some updates to clarify this once I understand whats going on.

[chelseacarter29]

Hey @oidebrett - when you do the data load, the 'site' using the example below is 'Behind-the-Tech'

python -m data_loading.db_load https://feeds.libsyn.com/121695/rss Behind-the-Tech

This then is populated in what we usually refer to as the 'site switcher' in the demo UI.

If you leave the site switcher set to 'all' it will ask all of the sites, but of course you can filter this down. Likewise, you can filter this in the code by editing this in the config_nlweb file - starting on line 4, you can see where you could replace 'yoursite' with 'Behind-the-Tech'

# Comma-separated list of allowed sites, if you have multiple sites
# in your store and want to restrict search to some sites. Defaults 
# to all, for now.
#sites: "yoursite"

Hope that helps! Let me know if more questions. (And sorry for the delay in getting back to you)

[oidebrett]

Thank you for you response. I can see from your demos that your explanation should work. However, in my case, I cant get any results returned if I dont specifically set the "Site Switcher" to "Behind-The-Tech". I am using qdrant_url as the vector db.

I have loaded the qdrant db with

python -m data_loading.db_load https://feeds.libsyn.com/121695/rss Behind-the-Tech

I can see my embeddings in my vector db and like I said I can get the results returned from nlweb when I specifically set the "site switcher".

However, if I leave the "site switcher" to "All", I get no results and I see that NLWeb is using the WhoRetriever to find but its filtering in the qdrant search query with

'site'='nlweb_sites'

then no vectors are returned.

Hopefully I am clear :)