How to handle client side redirects and not HTTP redirects?

[Olliejp]

Hi guys.

I was wondering if there was a way to handle client side redirects? For example if I try to run this code:

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig


async def scrape_single_page(url: str):
    browser_config = BrowserConfig(headless=True, verbose=False)
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.DISABLED,
        magic=True,
        remove_overlay_elements=True,
        page_timeout=12000,
        verbose=False
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        results = await crawler.arun(url, config=run_config)
    
    return results

x = asyncio.run(scrape_single_page("https://www.simpleenergy.com/"))
print(x.url)
print(x.redirected_url)
print(x.status_code)

I get status code 200, and the redirected url is not the actual redirected url in the browser directed client side... Ideally I would like to access the true final url which is https://uplight.com/.

Thanks for your help.

[Olliejp]

Just FYI, sorry for the spam, I seem to have been able to fix by adding:

wait_until="networkidle"

To run_config.