Response status: 404 on many request

[Sontranwakumo]

Thanks for the great project!

Lately, I have had issues when fetching profile, getUserIdByScreenName, getTweet, all of them have encoutered the same error: Response status: 404

About a week ago, this problem have never appeared.
I updated latest version and still getting the same issue, I guess that Twitter have updated these API endpoints.

[videvian]

We see the same issue. None of the requests succeed now and fail both locally and on deployed environment for resolving profile by the user name getProfile.

I've noticed that for getProfile this endpoint is specified in code: https://api.x.com/graphql/96tVxbPqMZDoYB5pmzezKA/UserByScreenName. But Twitter is using https://x.com/i/api/graphql/ZHSN3WlvahPKVvUxVQbg1A/UserByScreenName. I tried to use this URL locally but I get this error (we don't use credentials to log in):

AuthenticationError: {"errors":[{"message":"Sorry, that page does not exist","code":34}]}
    at TwitterGuestAuth.updateGuestToken (/Users/.../node_modules/@the-convocation/twitter-scraper/src/auth.ts:259:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async TwitterGuestAuth.installTo (/Users/.../node_modules/@the-convocation/twitter-scraper/src/auth.ts:173:7)
    at async requestApi (/Users/.../node_modules/@the-convocation/twitter-scraper/src/api.ts:63:9)
    at async getProfile (/Users/.../node_modules/@the-convocation/twitter-scraper/src/profile.ts:192:15)
    at async Scraper.getProfile (/Users/.../node_modules/@the-convocation/twitter-scraper/src/scraper.ts:126:17)

Curious if it's the same issue now as recently it was with 403. We tried adding cycletls and it didn't help.

[vmo0t2xw00t]

Getting the same error as of a day or so ago.

[cjmaxik]

All the GraphQL IDs were changed. Also, the requests now require User-Agent header present.

[karashiiro]

Updated most of the GQL endpoints just now (v0.19.1), there are a bunch of test failures on the user auth test suite I haven't figured out yet, though. Still, this should resolve the 404s, at least. I also added user-agent to the installed headers in the header update methods, just to be sure it's passed consistently.

[MoralCode]

Could this be related to https://devcommunity.x.com/t/deprecation-of-the-1-1-guest-activate-endpoint-in-the-twitter-api/200591?

Seems like this endpoint has been deprecated since 2023 and maybe just got removed?

its used in the code here:

twitter-scraper/src/auth.ts

Line 245 in abbd77f

const guestActivateUrl = 'https://api.x.com/1.1/guest/activate.json';

[karashiiro]

That seems unlikely, or else it wouldn't be possible to view tweets unauthenticated anymore (unless they introduced a new equivalent, of course).

[MoralCode]

#166 fixed my issue (i was seeing the "that page does not exist" code 34 error as others here)!

Edit: its back
Edit edit: ok it seems to be intermittent

[videvian]

I still get {\"errors\":[{\"message\":\"Sorry, that page does not exist\",\"code\":34}]} with v0.19.1 from TwitterGuestAuth.updateGuestToken

[mickael-btc]

It happens sometimes when the header "x-client-transaction-id" is missing.
I use this lib to generate it: https://github.com/Lqm1/x-client-transaction-id
Seems like they check more and more on headers, "x-xp-forwarded-for" might be the next one to be required