feat(agents): add shared URL policy for web tools

[ben-reitz]

Summary

Adds the URL policy that web_fetch (next PRs) checks on every hop, and moves abortable() out of websearch so both web tools share it. Internal only; nothing is exported from the package yet.

packages/agents/src/web/
├── url-policy.ts      # validateUrl, validateRedirect, validateUrlPolicy, isPrivateOrLocalHost
├── abortable.ts       # moved from search/source.ts; race a promise against an AbortSignal
└── search/            # now imports ../abortable

The policy relies on the WHATWG URL parser having already canonicalised the hostname, so there is no hand-written IP parsing:

validateUrl(input, policy)
  new URL(input)                        # rejects malformed; canonicalises host
  scheme ∉ {http, https} → disallowed
  userinfo present      → disallowed
  length > maxUrlLength → invalid
  host matches blockedHosts → disallowed
  allowedHosts set and host ∉ allowedHosts → disallowed
  !allowPrivateHosts and isPrivateOrLocalHost(host) → disallowed
  drop fragment; return canonical URL

isPrivateOrLocalHost is a range table — a deliberate subset of the IANA IPv4/IPv6 Special-Purpose Address Registries (RFC 6890): loopback, RFC 1918, CGNAT, link-local (cloud metadata), 192.0.0/24, 198.18/15, multicast, 240/4; IPv6 ::/96 (incl. ::, ::1), IPv4-mapped (re-checked as v4), NAT64, 6to4, fc00::/7, fe80::/10, fec0::/10, ff00::/8; plus localhost, *.localhost, *.internal. Hostnames are not resolved — 127.0.0.1.nip.io passes. In production Workers egress cannot reach private ranges anyway; this exists for local dev, other runtimes, and defence in depth, and the header comment says so.

Evidence

• Before: no shared policy; Think's fetch_url carried its own isBlockedHost list (replaced in refactor(think): share the private-host policy with agents/webfetch #2521).
  After: src/tests/web/url-policy.test.ts — one case per class, including the spellings that prove we lean on the parser:

  validateUrl("http://2130706433/")        → disallowed (127.0.0.1)
  validateUrl("http://0x7f.1/")            → disallowed (127.0.0.1)
  validateUrl("http://[::ffff:127.0.0.1]/")→ disallowed ([::ffff:7f00:1])
  validateUrl("http://169.254.169.254/")   → disallowed
  validateUrl("https://docs.example.com/", { allowedHosts: ["*.example.com"] }) → ok
  validateRedirect(from https://a, to http://127.0.0.1/) → disallowed_redirect
  

  vitest run src/tests/web → 101 passed; typecheck → 132 projects.

Merge Danger

Door: two-way

Internal module with no consumers until #2527.

Blast Radius: none

abortable was never exported from agents/websearch, so the move is invisible to users. websearch tests unchanged and passing.

[changeset-bot]

⚠️ No Changeset found

Latest commit: c9ec2dc

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[agent-think]

✅ agents import sizes: no significant changes (27732c63 → c9ec2dcb, workflow run)

[pkg-pr-new]

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@2526

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@2526

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@2526

hono-agents

npm i https://pkg.pr.new/hono-agents@2526

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@2526

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@2526

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@2526

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@2526

commit: c9ec2dc