Can't remove default headers. Can't overwrite `"sec-fetch-mode"`, `"connection"` headers.

[AlttiRi]

• I can't remove any default header.
• Setting "connection": "keep-alive" throws an error.
• I can't overwrite default "sec-fetch-mode" header correctly.

1. I can't remove the default headers

I can't remove the default headers: host, connection, accept, accept-encoding, accept-language, sec-fetch-mode, user-agent.

For example, curl allows to remove any header (even host header):
curl http://127.0.0.1:3000 -H 'User-Agent:' -H 'Accept:' -H 'Host:'
And yes, it works.

Using of "" will send "" as the header's value, using of null will send "null" as the header's value.

2. "connection": "keep-alive"

Also, if I manually set connection: "keep-alive" I get the error (code: 'UND_ERR_INVALID_ARG').

const resp = await fetch(url, {
    headers: {
        "connection": "keep-alive"
    }
});

It's important for code compatibility reason — since node-fetch uses connection: close by default.

3. "sec-fetch-mode"

It uses "sec-fetch-mode": "cors" by default.

As I said above, I can't remove this header (as well as any other default header).

More over, I can't change it correctly.

For example, using of:

await fetch(url, {
    headers: {
        "sec-fetch-mode": "navigate"
    }
});

Will produce a request with sec-fetch-mode: navigate, cors, while it should be sec-fetch-mode: navigate.

---

"undici": "4.16.0"

[ronag]

Can you check what Chrome does? I think most of the behavior you are seeing is regulated by the spec.

[AlttiRi]

2 and 3 are obviously bugs.

For 1, for me:

• Either add a way to remove the default headers, for example, by setting its value to null (not "null"). It's the best choose especially if it will be a part of Node.js. (UPD: However it will not be compatible with native fetch, so no.)
• Or at least remove sec-fetch-mode from the default headers (and maybe accept-language too).
  (For me it would be enough. Why you have even added it to default?)

---

Can you check what Chrome does? I think most of the behavior you are seeing is regulated by the spec.

What does Chrome have to do with it? Chrome, as a browser, also follows CORS limitations, for example.
But even Chrome:

• Does not throws errors if there is "connection": "keep-alive" in headers of fetch.
• Correctly set any custom sec-fetch-mode header value (with Web Extension API).

HTTP specification does not say that these headers are mandatory.
But undici forces to use them without any way not to do.

---

So, removing sec-fetch-mode from default headers is most simple and the correct thing.

If someone want to emulate a browser's behaviour he will add both sec-fetch-mode and sec-fetch-dest, sec-fetch-site: none, sec-fetch-user headers. As well as sec-ch-* headers.

Why a tool that going to be a part of Node.js have too much own vision of which headers are mandatory to use?

[ronag]

2 and 3 are obviously bugs.

For 1, for me:

• Either add a way to remove the default headers, for example, by setting its value to null (not "null"). It's the best > choose especially if it will be a part of Node.js. (UPD: However it will not be compatible with native fetch, so no.)
• Or at least remove sec-fetch-mode from the default headers

https://fetch.spec.whatwg.org/#http-network-or-cache-fetch

Step 13

Append the Fetch metadata headers for httpRequest. [FETCH-METADATA]

https://w3c.github.io/webappsec-fetch-metadata/#sec-fetch-mode-header

(and maybe accept-language too).

https://fetch.spec.whatwg.org/#fetching

Step 13:

If request’s header list does not contain Accept-Language, then user agents should append (`Accept-Language, an appropriate header value) to request’s header list.

---

What does Chrome have to do with it? Chrome, as a browser, also follows CORS limitations, for example.

We try to match the behaviour of Chrome. Basically we use it as our reference implementation.

• Does not throws errors if there is "connection": "keep-alive" in headers of fetch.

That is something we might want to revisit. It just don't make much sense to provide that header. Why do you do that? Depending on the settings of the undici client it might or might not be correct and might need to be stripped anyway if keep alive is disabled on the connection.

• Correctly set any custom sec-fetch-mode header value (with Web Extension API).

I'm not sure why you think you can't do that today?

HTTP specification does not say that these headers are mandatory.
But undici forces to use them without any way not to do.

I don't quite agree with this. You can disable the keep-alive header by disabling keep-alive and the other headers are mandated by the spec, as for as I know at least.

[ronag]

Sorry I accidentally edited your response instead of quoting it. Sorry. Tried to restore it.

[AlttiRi]

So, what about "sec-fetch-mode"?

It's a pure browser's thing, like CORS.
Why Node.js aimed undici uses it by default (and there is no way to disable it)?

---

Why do you do that?

I have already written it:

It's important for code compatibility reason — since node-fetch uses connection: close by default.

And because it's a pure bug — I set the correct, valid HTTP header (even it used by default), but undici fails. Also it works without errors in the browser and in any other HTTP client.

---

I'm not sure why you think you can't do that today?

Because of the bug №3:

await fetch(url, {
   headers: {
       "sec-fetch-mode": "navigate"
   }
});

Will produce a request with sec-fetch-mode: navigate, cors, while it should be sec-fetch-mode: navigate.

[AlttiRi]

You can disable the keep-alive header by disabling keep-alive and the other headers

Wait.
Could you show me the example?

[ronag]

You can disable the keep-alive header by disabling keep-alive and the other headers

Wait. Could you show me the example?

setGlobalDispatcher(new Agent({ pipelining: 0 }))

[ronag]

Regarding the "sec-fetch-mode": "navigate"... as far as I can tell the spec doesn't allow it and neither does Chrome. We could bypass the spec to make more sense in a node environment but I'm not familiar enough with these things to have confidence in making a good decision. Maybe @annevk can provide some guidance?

[ronag]

Actually we were doing it slight wrong, we should set, not append the header. 5b9acf2

[ronag]

One way to do what you want would be:

await fetch(url, {
   mode: 'navigate'
});

But again... the spec explicitly disallows it. But maybe it would make sense to bypass that? I don't know.

[AlttiRi]

setGlobalDispatcher(new Agent({ pipelining: 0 }))

It just changes connection: keep-alive to connection: close.

It does not remove any default header.

---

neither does Chrome.

As I already said, Chrome allows to do it with web extension.

Why it's not possible to do from a web page context? Because it's a browser. Web browsers have a lot of limitations.

Browsers do not allow:

• CORS,
• CORB,
• changing/removing "sec-fetch-*", <--
• reading "cookies" HTTP header
• and other things.

undici is not a browser. Don't overengineer it.

[ronag]

It does not remove any default header.

We need to set one of the headers. I'm not sure what you want here? #1307 PR welcome.

As I already said, Chrome allows to do it with web extension.

How? I'm curious. Can you show an example. I'm not familiar with web extension.

EDIT: Ah, yea doing it with web extensions is just a hole to get around the spec.

[ronag]

undici is not a browser. Don't overengineer it.

Easier said than done. Knowing what parts of the spec to ignore or not is not that easy. To keep things simple with the limited development time we have (I'm doing this work for free) it's much easier to just follow the spec verbatim, i.e. the aim here is to match the browser as closely as possible and not bike-shed over what is right or wrong; the spec/Chrome is right (with few well discussed exceptions).

undici is not a browser. Don't overengineer it.

fetch is a browser API. If you don't want/need the browser behavior then don't use the fetch API. We have much better API's for node, e.g. undici.request.

[AlttiRi]

ModHeader allows to edit and remove entirely any header (except "host", "connection", "cache-control" (on a page reload), "pragma" (on "Disable cache" with DevTools)).

---

fetch is a browser API.

So, I assume the next things will be implementing of CORS, CORB limitations, restricting of the access to "cookie" header since it's a browser API? [sarcasm]

---

I want to use Fetch API because it's well know and pretty convenient API.

Since it's a Node.js library I expect that it will have no limitations/additional headers which browsers applies due to security reason. In particular, no sec-fetch-* headers.

---

Or just add a way to remove the default headers.

With some option for setGlobalDispatcher, for example.

---

curl http://example.com/ -H 'User-Agent:' -H 'Accept:' -H 'Host:' no one header, but it works (However, with 400 - Bad Request error)

curl https://example.com/ -H 'User-Agent:' -H 'Accept:' only host header and it works with no error.