Is the 4chan boards scraping working correctly? Dataset status not updating

[maysamazzon]

Hi everyone,

I’ve been trying to scrape data from 4chan boards for a few days now, but the Dataset Status never changes. I’ve followed all the instructions and everything seems to be running fine, but the status remains the same.

I’ve also tried several solutions, but nothing has worked so far.

Does anyone know if the scraper for 4chan boards is currently working properly? Any help or guidance would be much appreciated!

Thanks in advance.

[sal-uva]

Hi @maysamazzon, the "Dataset status" box is for when you're running a query. Are you correctly collecting data, and did you generate the sphinx indexes? If you go to Control Panel and check the Active Workers box, there should be "fourchan-board" / "fourchan-thread" workers there.

4chan is still supported but at the moment we're working on a different system for local data sources, which we hope to launch in the coming months.

[maysamazzon]

Hi @sal-uva thanks for replying! But I could not correctly collect data, and could not generate the sphinx indexes because it keeps loading the query. I have already tested connection and it seems to be working just fine, but the results won't show up.

About the other system you're working on, it will definitely be helpful for us the same way 4cat has been.

[dale-wahl]

Just for clarity: these additional steps need to be followed to enable local datasources (4chan or 8kun). Your logs (e.g., in the Control Panel -> View Logs) should report the boards you are collecting data from if successfully set up.

[maysamazzon]

I've reinstalled a few times without success. Even without queries, scrap and write information appears in the logs. When I query the database directly, I see that it's saving queries I didn't make to the table (it simply grabs all the data and writes it), but queries I make on the front-end never appear. I can't imagine what I'm doing wrong.

[dale-wahl]

Perhaps you can provide some screenshots and/or snippets of the logs so we can better diagnose.

There are two processes. You setup the scraper (following the instructions I linked to), which it sounds like may be working. While 4CAT is running, that scraper will collect the boards you enabled and the posts on them (since the data from those sources is ephemeral). Then, via the 4CAT frontend, you can query the data that scraper has already collected and stored on your 4CAT instance. E.g., you run the scraper on 4chan's p board for a week, then you query via the frontend for posts containing text about a specific topic within that week.