Large Tagset Issues - Feature Request + Bug Report

[drmstwizard]

There are two primary issues that come up while running the tag replacer on large tagsets (>500 uses). One is that Tumblr throws back a softblock with the following error code:

{"status":500,"statusText":"Internal Server Error","body":{"body":{"meta":{"status":429,"msg":"Limit Exceeded"},"response":[],"errors":[{"title":"Limit Exceeded","code":5029,"detail":"Your IP, XXX.XXX.XXX.XXX, has been blocked for excessive request volume. Please contact Tumblr Support at https://www.tumblr.com/support, including this IP in the message, if you believe this was in error."}]}}}

Too many softblocks, and the user's account is terminated (see Issue from Jan 30 of 2021 by another user!).

The other problem is that the code times out while fetching instances in especially large tagsets (>1000 uses) and just... logs out of Tumblr/restarts the app.

Given both of these issues, maybe there's a way to have the code fetch a maximum of 499 instances of a given tag with one fetch request? Having to run the app multiple times for large tags but decreasing the risk of an account termination or app failure would be a positive tradeoff! :)

[cubeghost]

Definitely aware of this! I'm working on rearchitecting things right now so that I can better rate limit the internal API requests made to Tumblr. Another thing I'm aiming for is being able to send progress updates to the front end (with websockets) so that it's not just waiting on one extremely long-running HTTP request.

I'm not super clear on the relationship between rate limiting and accounts being flagged, there have only been a few anecdotal reports over the years about accounts getting terminated and I haven't been able to get a lot of details from those.

When you say >500 uses, do you mean 500 posts or more like 500 pages?

[drmstwizard]

More than 500 posts using the tag. I've been terminated a few times for using the xkit tag replacer and have talked to @staff about what caused it, so I might be able to answer some questions -- I haven't spoken to them about it since 2019 though so my answers may be very out of date. From what I know from back then and what I saw in the softblock error today, it takes about 500 posts being replaced at once/in the same fetch for it to flag the automation for a softblock. Scraping for the tag doesn't flag anything, it's the actual replacement that Tumblr can sometimes flag as "bot behavior." I'm trying to think if I know anything else relevant, but I'm not a programmer so I'm not sure! I hope any of that is helpful!

…

On Tue, Dec 28, 2021, 10:01 PM alex baldwin ***@***.***> wrote: Definitely aware of this! I'm working on rearchitecting things right now so that I can better rate limit the internal API requests made to Tumblr. Another thing I'm aiming for is being able to send progress updates to the front end (with websockets) so that it's not just waiting on one extremely long-running HTTP request. I'm not super clear on the relationship between rate limiting and accounts being flagged, there have only been a few anecdotal reports over the years about accounts getting terminated and I haven't been able to get a lot of details from those. When you say >500 uses, do you mean 500 posts or more like 500 pages? — Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG5PDI6X27I7N55AQVH4RFDUTKB2NANCNFSM5K4YH57A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[whatdoyoumeanitsnotawesome]

has there been any movement on this?

if i might make a suggestion, maybe limit fetching results to the first 1,000 returns? sure, if you have hundreds of pages of results for a tag ([cough]likeido[cough],) that's really repetitious, but it's better than no results at all.