Skip to content

[Cors Proxy] Support Transfer-Encoding: Chunked - #2077

Merged
adamziel merged 2 commits into
trunkfrom
proxy-support-chunked-encoding
Dec 13, 2024
Merged

adamziel merged 2 commits into
trunkfrom
proxy-support-chunked-encoding

Conversation

@adamziel

Copy link
Copy Markdown
Collaborator

Adjusts the CORS Proxy to relay chunked responses using chunks.

Implementation

Some APIs use fixed content-length and respond with a predictable response body. Other APIs respond with chunks.

Before this PR, whenever the CORS proxy received a chunked response, it would only relay the Transfer-Encoding: Chunked header without actually transmitting the response body as chunks.

Chunks are an essential HTTP 1.1 feature that cannot be disabled, We also can't just filter out that header in the proxy response as the client expects either a Content-Length or body chunks, and we may not know the Content-length upfront.

With this PR, the CORS proxy outputs its response using the chunked encoding whenever the remote endpoint streams its response using the chunked encoding.

Testing

Adjusts the CORS Proxy to relay chunked responses using chunks.

 ## Implementation

Some APIs use fixed content-length and respond with a predictable
response body. Other APIs respond with chunks.

Before this PR, whenever the CORS proxy received a chunked response, it would
only relay the `Transfer-Encoding: Chunked` header without actually transmitting
the response body as chunks.

Chunks are an essential HTTP 1.1 feature that cannot be disabled, We
also can't just filter out that header in the proxy response as the
client expects either a Content-Length or body chunks, and we may not
know the Content-length upfront.

With this PR, the CORS proxy outputs its response using the chunked encoding
whenever the remote endpoint streams its response using the chunked encoding.

 ## Testing

* Run `nx start playground-php-cors-proxy`
* Fixed-length response: Go to http://127.0.0.1:5263/cors-proxy.php?https://raw.githubusercontent.com/WordPress/wordpress-playground/5e5ba3e0f5b984ceadd5cbe6e661828c14621d25/README.md
* Confirm it loaded the file
* Chunked encoding: Go to http://127.0.0.1:5263/cors-proxy.php?https://api.wordpress.org/plugins/update-check/1.1/
* Confirm the response says the word `error`
* Error handling: Go to http://127.0.0.1:5263/cors-proxy.php?https://nosuchsite.playground.wordpress.net
* Confirm the response says Bad Request Hostname could not be resolved
header('Access-Control-Allow-Credentials: true');
header('Access-Control-Allow-Methods: GET, POST, OPTIONS');
header('Access-Control-Allow-Headers: Accept, Authorization, Content-Type, git-protocol');
header('Access-Control-Allow-Headers: Accept, Authorization, Content-Type, git-protocol, wp_blog, wp_install');

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Casual addition – these two headers are used by WordPress core when requesting api.w.org

@adamziel
adamziel merged commit 85c5d30 into trunk Dec 13, 2024
@adamziel
adamziel deleted the proxy-support-chunked-encoding branch December 13, 2024 08:44
edisplay pushed a commit to edisplay/wordpress-playground that referenced this pull request Dec 17, 2024
…he latest release (WordPress#2056)

Fixes a regression created in WordPress#2077. The minified nightly build
accidentally became a minified build of the latest WordPress version.

In the multitude of resolveWordPressRelease() calls used in building the
minified WordPress releases, the last call in the chain started
transforming the nightly URL into a 6.7.1 release URL. This PR removes
that one final call, and also simplifies the Playground CLI build
pipeline.

 ## Testing instructions

Apply this PR, switch WP version to nightly, confirm in wp-=admin it's
indeed the nightly release
adamziel added a commit that referenced this pull request Jan 2, 2025
Adds support for `transfer-encoding: chunked` on Apache, Nginx, and other web servers.

#2077 added support for `transfer-encoding: chunked` in a way that works on a local PHP dev server. The proxy manually chunks the output bytes and outputs each chunk's header, separator, body, and trailer.

However, it doesn't work on playground.wordpress.net because the web server there handles the chunked encoding on its own. The bytes echoed by cors-proxy.php are treated as body bytes, which messes up the response body.

This PR restricts the manual chunking to a local CLI dev server.

## Testing instructions

Run cors-proxy.php in Apache or so and confirm that requesting resources served with chunked encoding works with this patch but not without it. One such URL is https://adamadam.blog/feed/.
adamziel added a commit that referenced this pull request Jan 9, 2025
#2114)

Adds support for `transfer-encoding: chunked` on Apache, Nginx, and
other web servers.

#2077 added support for `transfer-encoding: chunked` in a way that works
on a local PHP dev server. The proxy manually chunks the output bytes
and outputs each chunk's header, separator, body, and trailer.

However, it doesn't work on playground.wordpress.net because the web
server there handles the chunked encoding on its own. The bytes echoed
by cors-proxy.php are treated as body bytes, which messes up the
response body.

This PR restricts the manual chunking to a local CLI dev server.

## Testing instructions

Run cors-proxy.php in Apache or so and confirm that requesting resources
served with chunked encoding works with this patch but not without it.
One such URL is https://adamadam.blog/feed/.
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 27, 2026
The proxy stopped relaying Content-Length in #2077, which restructured
the header callback to support chunked responses. Relay it again for
fixed-length responses, deciding after all headers arrive because HTTP
ignores Content-Length when Transfer-Encoding: chunked is present.

With Content-Length relayed, clients detect responses cut short by a
target failure, and HEAD becomes useful for learning a file's size
before requesting ranges. Accept HEAD, skip the response size cap for
it since no body is relayed, and tell curl not to wait for a body.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
brandonpayton added a commit that referenced this pull request Sep 28, 2026
…sponses (#4365)

## Motivation for the change, related issues

The CORS proxy has not relayed the target's `Content-Length` since
#2077. That PR restructured the response header callback to support
chunked responses, and every special-cased header gained an early
return. #2077 does not discuss dropping `Content-Length`, and the proxy
relayed it before that. Without it:

- Clients can't detect a response cut short by a target failure. The
body just ends early.
- Full responses carry no size. Playground's blueprint download progress
falls back to a 5MB estimate, for example.
- `HEAD` would be useless even if allowed. The proxy rejects it with 405
today.

`HEAD` plus `Content-Length` is the usual way for a client to learn a
file's size before reading ranges, e.g. to locate a ZIP archive's
central directory.

This PR builds on #4362, which added range request support and runs the
e2e tests from `test.sh`, and #4364, which stops relaying server-control
headers. Both have merged, and this PR is based on `trunk`.

## Implementation details

- **Relay `Content-Length` for fixed-length responses.** The header
callback holds the value and emits it when the header block ends, but
only if the response has no `Transfer-Encoding`. RFC 9112 §6.3 says
`Transfer-Encoding` overrides `Content-Length`, and headers can arrive
in either order. The held value resets for each header block, e.g. after
a `100 Continue`, and only digit-only values are relayed.
- Chunked responses are unchanged and still relayed as chunks.
- **Support `HEAD`.** Accept it in the method check, list it in `Allow`
and `Access-Control-Allow-Methods`, and set `CURLOPT_NOBODY` so curl
doesn't wait for the announced body. `HEAD` is a CORS-safelisted method,
and `Content-Length` is a safelisted response header, so browsers need
no preflight or exposure for this.
- `HEAD` also asks the target for `Accept-Encoding: identity`, as ranged
requests do. Clients use the `HEAD` size to compute range offsets, so it
must describe the same unencoded bytes the ranges address.
- The response size cap limits relayed bodies, so it no longer applies
to `HEAD`. A `HEAD` on a multi-GiB file returns its size instead of
`413`.
- When the proxy replaces a response with its own `413` or `502`, the
relayed `Content-Length` is removed with the rest of the target's
headers, and the held value is cleared.
- **Fix interim 1xx responses (existing bug).** The proxy took its
status from the first status line curl reported and relayed every header
block. A target that sent `103 Early Hints` or `100 Continue` before its
final response made the proxy answer with that interim status and
headers, followed by the final body, which clients reject. The proxy now
sets its status from the final status line and skips interim header
blocks entirely. A `101 Switching Protocols` isn't skipped: the proxy
can't relay another protocol, so it answers with `502` instead of
presenting the other protocol's bytes as a response body.
- The request body size check now applies only to `POST`, the only
accepted method with a body. Previously any non-`GET` method read
`$_SERVER['CONTENT_LENGTH']`.

**Effects inside Playground:** PHP.wasm's outbound HTTP
(`tcp-over-fetch-websocket.ts`) strips `Content-Length` before handing
responses to PHP, so PHP sees no difference there. `HEAD` requests from
PHP that go through the proxy, e.g. `wp_remote_head()`, now succeed
instead of getting `405`. When the client receives the target's
`Content-Length`, a proxied body cut short now makes `fetch()` fail
instead of returning short data. That doesn't hold for chunked
responses, or when a server in front of PHP compresses the response and
drops `Content-Length`; the README says so. Blueprint resource downloads
and URL-based file reading gain real sizes for progress. When a target
compresses a response, `Content-Length` is the compressed size, so a
download step's progress can show complete before the download ends.
That's cosmetic, since the overall operation can't finish early, and it
already happens with the 5MB fallback.

## Testing Instructions (or ideally a Blueprint)

```bash
cd packages/playground/php-cors-proxy
bash test.sh
```

Expect PHPUnit to pass, followed by `All 66 assertions passed.` from the
e2e script. New and updated e2e tests:

- A fixed-length response relays `Content-Length`.
- A chunked response relays `Transfer-Encoding: chunked` but not
`Content-Length`, even when the target sends `Content-Length` before
`Transfer-Encoding: chunked`. A small raw-socket server,
`tests/e2e/raw-upstream-server.php`, produces that response, because
PHP's built-in server can't send chunked responses. I confirmed this
test fails if `Content-Length` is relayed as soon as it arrives, or
without the chunked check.
- `HEAD` on a 4 GiB file returns `200` with `Content-Length:
4294967296`, and `HEAD` asks the target for `Accept-Encoding: identity`.
- A target that sends `103 Early Hints` before its `200` produces a
`200` response with the final body and without the interim `Link`
header. The raw server also produces this response, and a `101 Switching
Protocols` followed by other bytes, which gets a `502` with only the
proxy's headers.
- A target that fails mid-body now gives the client an
incomplete-response error (`CURLE_PARTIAL_FILE`) instead of a
complete-looking short body. The test also checks that the partial body
holds only the target's bytes; the test helper now captures bodies
through a callback so failed transfers keep what arrived.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* The CORS proxy supports `HEAD` requests, returning response headers
without a body and requesting identity encoding.
* Target `Content-Length` is relayed only when the response is not
chunked. Interim response headers are skipped, and partial responses can
be detected using the announced length or, for range responses,
`Content-Range`.
* Protocol-switching responses are rejected with a 502 error.
Response-size limits do not apply to `HEAD` requests.
* **Documentation**
* Clarified supported methods and guidance for detecting truncated
responses.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

No open projects
Archived in project

Development

Successfully merging this pull request may close these issues.

1 participant