error code: 1101

[CoPost14]

Hello,

Sometimes I got an error responce (error code 1101) on requests. It seems totally random. Sometimes a retry a few seconds later works, but sometimes not. However, a day alter this request will be fine.
Example of today:
curl.exe -s "https://atnapi.juandomingo.net/athletes/profile/8879130" (today 18/3, 10:17)
After 3 retries I got a response.

Besides this, sometimes data using this is not up2date and it take something like 24hours before alle data is there. For a race some results of an athlete is there, but from another athlete it is not there yet. Any idea how this could be possible?

Many thanks!

regards,
Co

[zeskeertwee]

The 1101 error is caused by too fast scraping of the atletiek.nu website, which makes it hide results in the webpage it returns when it decides it is too fast, causing this error in the worker: panicked at /home/jasper/Code/atletiek-nu-api/atletiek-nu-api/src/models/athlete_profile.rs:145:54: called Result::unwrap()on anErr value: ParseIntError { kind: Empty }. That causes a 1101 error from cloudflare. Note that it is also possible that it is a bug in the scraper that causes it to fail, if you find this you should definitely report it (but this is not the case here)!

The second problem is due to some caching I had set up long ago, I have changed it now, it should now only cache from the worker side, no longer cloudflare in the middle. It will still do some caching, it is set to 1 hour validity. So after an hour it will re-do the request.

[CoPost14]

Thanks for improving the delay in the updates of the data by decreasing the caching time out. This helps a lot!
About the other issue, is there something I can do about this or something you can change? Maybe adding some delay in this part?

[zeskeertwee]

As far as I am aware there is nothing I can do to fix this issue, besides not hitting the site with a lot of requests in a very short timespan. I also do not know from what IPs the web requests from the workers to atletiek.nu originate (e.g. if they are always the same). I would suggest spreading the requests over a longer duration, if possible (mimicing more 'normal' browsing of the site).

The only real way to improve this would be to have many proxies so it can round-robin the requests from different IP origins.

[CoPost14]

That's a pitty, it was working perfect.... I have treid the local API again, also this gives errors now... same cause?

Checking for updates...
Checking target-arch... x86_64-pc-windows-gnu
Checking current version... v0.2.7
Checking latest released version... v0.2.7
No update is available.
Failed to load cache from disk: The system cannot find the file specified. (os error 2)
Creating empty cache
�[34m�[1;39mRocket has launched from �[4mhttp://127.0.0.1:8000�[0m�[34m�[0m
REQUEST 0 --> https://www.athletics.app/atleet/profiel/899027
REQUEST 0 --> STATUS 200 OK
Oops! It seems the application has crashed!
Would you like to submit a crash report?
(type 'y' to submit, 'n' to not submit, 'v' to view the crash report, and 'h' for help)
crash-reporter > y
Connecting to crash report server... Fatal error occured during crash report submission: Failed to connect to panic-report server!
�[1m>>�[0m �[31mHandler �[39mget_athlete_profile�[0m�[31m panicked.�[0m
�[1m>>�[0m �[33mA panic is treated as an internal server error.�[0m
�[1m>>�[0m �[33mNo �[1;34m500�[0m�[33m catcher registered. Using Rocket default.�[0m

[zeskeertwee]

Hm, in this case it seems that the error is from line 77 where it tries to select the div containing the name. The query seems to work in my inspect element so something must be going on here.

atletiek-nu-api/atletiek-nu-api/src/models/athlete_profile.rs

Line 77 in 407f437

let mut page_title_texts = html.select(&page_title_selector).next().unwrap().text();

Well, shit. This is the HTML it receives.

This will probably require some tricks to get around to make the whole thing work again.