[cliffhall]

Closes #5134

Description

fetch_url treated any response with no Content-Type header as HTML (or not content_type), so JSON, plain text, CSV and other header-less payloads were pushed through readability and markdownify: they came back altered (underscores and asterisks escaped, newlines collapsed without Node) or, for an empty body, as a simplification error.

A header-less body is now simplified only when the body itself opens like an HTML document: after any byte-order mark and whitespace, an <!doctype html or an <html tag, case-insensitively (new helper _body_looks_like_html). Anything else comes back raw, like any other non-HTML content type. Two small additions in the same expression: application/xhtml+xml counts as HTML, and the media type is matched case-insensitively (Text/HTML was previously returned raw). The existing "<html" in page_raw[:100] sniff is unchanged, and the doctype sniff applies only when the header is missing, so a declared non-HTML type is still honored.

Server Details

• Server: fetch (mcp-server-fetch)
• Changes to: the fetch tool's content handling (is_page_html in fetch_url); no tool, argument or prompt changes

Motivation and Context

See #5134. The reporter's prototype (and the branch linked there) sniffed the body the same way; this keeps the change to the HTML decision only.

How Has This Been Tested?

New protocol-level tests in src/fetch/tests/test_missing_content_type.py (13 cases) drive serve() through a ClientSession with HTTP mocked: header-less JSON object/array, plain text and CSV come back raw and unaltered; a header-less empty body no longer reports "Page failed to be simplified"; header-less HTML (upper/lowercase doctype, BOM plus whitespace, a <HTML> tag beyond the 100-character window) is still simplified; application/xhtml+xml and Text/HTML; charset=UTF-8 are simplified; a doctype does not override a declared text/plain. Eight of them fail against v2/main. Fetch coverage: server.py 100% lines, 98.4% branches. npm run local:gate exits 0.

Client evidence: a local HTTP server that sends no Content-Type header serving /data.json ({"snake_case_key": "*value*", "items": [1, 2, 3]}), /notes.txt (plain_text with *asterisks*\nsecond line) and /page (a <!DOCTYPE html> article with no <html> tag). The server ran as mcp-server-fetch --ignore-robots-txt --allow-private-ips; "before" is origin/v2/main's mcp_server_fetch source.

Inspector CLI 2.9.0, --method tools/call --tool-name fetch --tool-arg url=…, with --protocol-era legacy (2025-11-25) and --protocol-era modern (2026-07-28; result _meta carries io.modelcontextprotocol/serverInfo). Both eras returned identical text; exit 0 for all twelve calls.

URL	Before (v2/main)	After (this branch)
/data.json	{"snake\_case\_key": "\*value\*", "items": [1, 2, 3]} (markdown-escaped)	Content type cannot be simplified to markdown, but here is the raw content: then {"snake_case_key": "*value*", "items": [1, 2, 3]}
/notes.txt	plain\_text with \*asterisks\*	raw prefix, then plain_text with *asterisks* / second line
/page	simplified: This is a [link](https://example.com/x) …, ## Sub heading	identical (still simplified)

LLM client: Claude Code 2.1.311 (claude -p --strict-mcp-config, fetch server as above; this client does not let the era be chosen, so it is not labeled with one). Prompt: "Use the fetch tool to fetch http://127.0.0.1:51347/data.json. Reply with exactly two lines: first, the JSON value of the key snake_case_key exactly as the tool returned it …; second, yes or no: did the tool say the content type cannot be simplified and return raw content?"

• Before: "\*value\*" / no
• After: "*value*" / yes

Breaking Changes

None. No configuration change. A header-less non-HTML response is now returned raw instead of as (mangled) Markdown, which is the fix.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have read the MCP Protocol Documentation
• My changes follow MCP security best practices
• I have updated the server's README accordingly (not applicable: the README does not describe how content types are detected)
• I have added a changeset (npm run changeset) if this changes what a TypeScript server publishes (not applicable: fetch is a Python server)
• I have tested this with an LLM client
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have documented all environment variables and configuration options (not applicable: none added)

Additional context

Part of the Fetch v2.2.0 tracker #5265. The raw-content prefix reads Content type cannot be simplified… (an empty media type) when the header is missing; that line is unchanged here, since the issue is about the HTML decision.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: 593dab9

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[Copilot]

🟢 Approval recommended

The focused classification change addresses the reported defect with relevant regression coverage and no identified blocking issues.

No open issues 🎉

What changed in this PR

This PR fixes #5134 in the Fetch server by returning header-less non-HTML payloads unchanged instead of sending them through HTML simplification.

Changes:

• Adds an opening-body HTML check for responses without a Content-Type.
• Recognizes XHTML and case-insensitive HTML media types while preserving the existing <html sniff.
• Adds 13 protocol-level regression cases covering raw payloads, empty responses, HTML detection, and declared content types.

File	Description
src/​fetch/​tests/​test_missing_content_type.py	Adds regression coverage for content classification and returned content.
src/​fetch/​src/​mcp_server_fetch/​server.py	Replaces the missing-header HTML assumption with body detection and expands HTML media-type recognition.

_{Review effort: Balanced}

[cliffhall]

Copilot round 1: clean (no findings, no suppressed comments). The review loop ends here on the first clean round; nothing was pushed after it.

[cliffhall]

Pushed 593dab9 for the "Test fetch (Windows)" failure.

The bom-and-whitespace case crashed in extraction, not in this PR's detection: readabilipy 0.2.0 (what uv.lock pins) writes the page to a temp file in the locale encoding, and cp1252 cannot hold . That predates this PR (a header-less BOM page, or any text/html page with a non-cp1252 character, crashes the same way on v2/main), so it is filed as #5303. Upstream fixed it in readabilipy 0.3.0.

In test_missing_content_type.py:

• The protocol-level BOM case is kept but skipped on Windows, with a reason naming mcp-server-fetch: HTML extraction crashes with UnicodeEncodeError on Windows for non-cp1252 pages (readabilipy 0.2.0) #5303. No retries.
• A new test_body_detection_skips_bom_and_whitespace asserts _body_looks_like_html directly on every platform: a BOM before a doctype, before whitespace and a doctype, or before <HTML> is HTML, and a BOM before JSON or plain text, or an empty body, is not. So what mcp-server-fetch: missing Content-Type header forces non-HTML payloads into readability extraction, failing with error #5134 decides is still checked on Windows.

No change to server.py. npm run local:gate exits 0. I am requesting another Copilot round.

[Copilot]

🟢 Approval recommended

The narrowly scoped fix addresses the reported defect with focused regression coverage and no identified blocking issues.

No open issues 🎉

_{Review effort: Balanced}

[cliffhall]

Copilot round 2 (on 593dab9, the Windows test fix): clean, with no findings and no suppressed comments. The loop stops here on a clean round.