Proxy server

[lollita]

Do it is possibile to use a Proxy server or Socks server?

const scraper = new Scraper({ fetch: (url, options = {}) => {
	return fetch(url, {
		...options,
		agent: new HttpsProxyAgent("http://127.0.0.1:8080")
		//agent: new SocksProxyAgent('socks5h://127.0.0.1:1080')
	});
}});

[karashiiro]

You can use whatever you want, either by passing a request transformer, or by just passing in your own version of fetch like you've done here. I'm not sure if the built-in fetch in Node supports that, but if there's any library that does, you can just plug that in there too.

[lollita]

My example not work.
function requestApi() not use fetch.
How to use transformer instead?
a working example?

[karashiiro]

I don't know if this example still runs (backend probably needs to be updated) but the transformer syntax is the same, see here: https://github.com/the-convocation/twitter-scraper/blob/main/examples/react-integration/src/App.tsx

[karashiiro]

That example is just a request rewrite, but that transformer allows you to change/override any other request option, too.

[lollita]

also with transform not work:
{"error":"Cannot read properties of undefined (reading 'name')"}
when load https://api.x.com/1.1/account/verify_credentials.json

const customFetch = {
	transform: {
		request(input, init) {
			if (input instanceof URL) {
				input.protocol = 'http';
				input.host = `127.0.0.1:8080;
				return [input, init];
			} else if (typeof input === 'string') {
				const proxy = new URL(input);
			console.log(input, init);
				proxy.protocol = 'http';
				input.host = `127.0.0.1:8080`;
				//proxy.protocol = 'socks5';
				//input.host = `127.0.0.1:1080`;
				return [proxy, init];
			} else {
				throw new Error('Unexpected request input type');
			}
		},
	}
};
}
scraper = new Scraper(customFetch);

[karashiiro]

Can you share what your HTTP proxy is doing? I'm wondering if it's proxying back an unexpected response and causing the library to behave incorrectly indirectly.

Or, you could add a response transformer to console.log the response the library is receiving, as an easier way to see what's happening. Something simple like this:

response(res) {
	console.log(res);
	return res;
}

---

Also, in this branch of your request transformer:

} else if (typeof input === 'string') {
	const proxy = new URL(input);
	console.log(input, init);
	proxy.protocol = 'http';
	input.host = `127.0.0.1:8080`;
	//proxy.protocol = 'socks5';
	//input.host = `127.0.0.1:1080`;
	return [proxy, init];
}

you'd want to set proxy.host, not input.host.

[lollita]

HTTP proxy error: Socket closed
Response {
  size: 0,
  timeout: 0,
  [Symbol(Body internals)]: {
    body: PassThrough {
      _events: [Object],
      _readableState: [ReadableState],
      _writableState: [WritableState],
      allowHalfOpen: true,
      _maxListeners: undefined,
      _eventsCount: 2,
      [Symbol(shapeMode)]: true,
      [Symbol(kCapture)]: false,
      [Symbol(kCallback)]: null
    },
    disturbed: false,
    error: null
  },
  [Symbol(Response internals)]: {
    url: 'http://127.0.0.1:8280/1.1/account/verify_credentials.json',
    status: 502,
    statusText: 'Bad Gateway',
    headers: Headers { [Symbol(map)]: [Object: null prototype] },
    counter: 0
  }
}

input.host = 127.0.0.1:8080;
fixed to
proxy.host = `127.0.0.1:8080;

also if I use a external proxy server not nodejs.

[amitduwal]

@karashiiro do you have an expmple for fetch using proxy

[karashiiro]

The one I linked above from the examples folder works for me, I had to adjust it slightly but there didn't seem to be issues with the library before, just with the example itself (which has now been updated).