fix: block local/private URL scraping in scrape tool

[stablegenius49]

Summary

• block scrape_url requests to localhost, private IP ranges, and hostnames that resolve to local/private addresses
• reject non-HTTP(S) scrape targets and validate redirect destinations before following them
• keep the change scoped to the explicit URL-scraping tool used for user-requested page summaries

Fixes #949.

Testing

• ./node_modules/.bin/eslint src/lib/agents/search/researcher/actions/scrapeURL.ts
• ./node_modules/.bin/tsc --noEmit

---

Summary by cubic

Block scraping of localhost and private network URLs in the scrape_url tool to prevent SSRF and unsafe requests. Only HTTP(S) targets are allowed, with validated and limited redirects. Fixes #949.

• Bug Fixes
  • Block localhost, loopback, and private IP ranges (IPv4/IPv6), plus common local TLDs.
  • Validate DNS results and reject hostnames that resolve to local/private addresses.
  • Allow only HTTP(S); follow redirects manually with validation and a 5-hop limit.
  • Scope change to the scrape_url tool used for user-requested page summaries.

^{Written for commit b56ac10. Summary will update on new commits.}

[cubic-dev-ai]

3 issues found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/lib/agents/search/researcher/actions/scrapeURL.ts">

<violation number="1" location="src/lib/agents/search/researcher/actions/scrapeURL.ts:65">
P1: IPv4-mapped IPv6 filtering can be bypassed with hex/compressed forms (e.g. `::ffff:7f00:1`), allowing local/private SSRF targets.</violation>

<violation number="2" location="src/lib/agents/search/researcher/actions/scrapeURL.ts:133">
P1: DNS lookup errors are swallowed in the URL safety check, causing fail-open behavior where unvalidated URLs may still be fetched.</violation>

<violation number="3" location="src/lib/agents/search/researcher/actions/scrapeURL.ts:152">
P1: DNS pre-validation is decoupled from the actual fetch connection target, leaving a DNS rebinding TOCTOU SSRF bypass path.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P1: DNS pre-validation is decoupled from the actual fetch connection target, leaving a DNS rebinding TOCTOU SSRF bypass path.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/lib/agents/search/researcher/actions/scrapeURL.ts, line 152:

<comment>DNS pre-validation is decoupled from the actual fetch connection target, leaving a DNS rebinding TOCTOU SSRF bypass path.</comment>

<file context>
@@ -17,6 +28,147 @@ You should only call this tool when the user has specifically requested informat
+  redirectCount = 0,
+): Promise<Response> => {
+  const safeURL = await assertSafeScrapeURL(rawURL);
+  const res = await fetch(safeURL, { redirect: 'manual' });
+
+  if (res.status >= 300 && res.status < 400) {
</file context>

[cubic-dev-ai]

P1: IPv4-mapped IPv6 filtering can be bypassed with hex/compressed forms (e.g. ::ffff:7f00:1), allowing local/private SSRF targets.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/lib/agents/search/researcher/actions/scrapeURL.ts, line 65:

<comment>IPv4-mapped IPv6 filtering can be bypassed with hex/compressed forms (e.g. `::ffff:7f00:1`), allowing local/private SSRF targets.</comment>

<file context>
@@ -17,6 +28,147 @@ You should only call this tool when the user has specifically requested informat
+  }
+
+  if (normalized.startsWith('::ffff:')) {
+    return isBlockedIPAddress(normalized.slice('::ffff:'.length));
+  }
+
</file context>

[cubic-dev-ai]

P1: DNS lookup errors are swallowed in the URL safety check, causing fail-open behavior where unvalidated URLs may still be fetched.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/lib/agents/search/researcher/actions/scrapeURL.ts, line 133:

<comment>DNS lookup errors are swallowed in the URL safety check, causing fail-open behavior where unvalidated URLs may still be fetched.</comment>

<file context>
@@ -17,6 +28,147 @@ You should only call this tool when the user has specifically requested informat
+        `Refusing to access local or private network URL: ${rawURL}`,
+      );
+    }
+  } catch (error: any) {
+    if (
+      error instanceof Error &&
</file context>

Suggested change

	} catch (error: any) {
	if (
	error instanceof Error &&
	error.message.startsWith(
	'Refusing to access local or private network URL:',
	)
	) {
	throw error;
	}
	} catch (error: any) {
	if (
	error instanceof Error &&
	error.message.startsWith(
	'Refusing to access local or private network URL:',
	)
	) {
	throw error;
	}
	throw new Error(
	`Unable to verify URL safety via DNS lookup for ${rawURL}: ${error instanceof Error ? error.message : String(error)}`,
	);
	}