LyricFind provider integration tests failed.

[skuill]

Looks like LyricFind has updated their site. We need to make sure that HtmlAgility returns ParsedText so far. If not, then switch to NetHttpClient.

[skuill]

The 403 Forbidden error often occurs when the server blocks requests that do not align with its security policies. In the context of web scraping, this is commonly due to missing expected headers (e.g., User-Agent) or IP-based blocking. The website might have implemented additional measures to prevent automated requests.

Potential Solutions

1. Add a User-Agent Header

Many servers block requests with missing or suspicious User-Agent headers. Specify it explicitly:

var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36");

string htmlPageBody;
try
{
    htmlPageBody = httpClient.GetStringAsync(uri).GetAwaiter().GetResult();
}
catch (HttpRequestException ex)
{
    _logger?.LogWarning($"HttpClient GetStringAsync threw an exception for uri: {uri}. Exception: {ex}");
    return string.Empty;
}

---

2. Introduce Delays Between Requests

If the server detects automated activity, it may temporarily block requests. Add a small delay:

await Task.Delay(1000); // 1-second delay

---

3. Add Additional Headers

Some sites validate headers to ensure requests mimic a real browser. Include headers like Referer and Accept:

httpClient.DefaultRequestHeaders.Add("Referer", "https://lyrics.lyricfind.com/");
httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
httpClient.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.5");

---

4. Use Proxy Servers

If the website blocks your IP address, consider using a proxy:

var httpClientHandler = new HttpClientHandler
{
    Proxy = new WebProxy("http://your-proxy-address:port"),
    UseProxy = true
};
var httpClient = new HttpClient(httpClientHandler);

---

5. Emulate a Full Browser

If simpler solutions fail, you can use a library like PuppeteerSharp or Playwright to emulate browser behavior, bypassing many server-side checks:

var browserFetcher = new BrowserFetcher();
await browserFetcher.DownloadAsync(BrowserFetcher.DefaultChromiumRevision);
var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = true });
var page = await browser.NewPageAsync();
await page.GoToAsync("https://lyrics.lyricfind.com/lyrics/parkway-drive-idols-and-anchors");
var content = await page.GetContentAsync();

---

Recommendations

Start with the simpler steps:

1. Add a User-Agent, Referer, and other headers.
2. If the problem persists, implement proxies or browser emulation.

[skuill]

In this case, the WAF (Web Application Firewall) protection mechanism requires JavaScript to be executed to generate a token. WAF token (aws-waf-token) generation occurs between a request returning 202 Accepted and a subsequent successful request. To implement this in C#, you'll need to use browser emulation or interact with JavaScript. One popular tool for this is Playwright.NET.