Access to the HTTP trailer

[tyoshino]

The trailer semantics in RFC 7230 (and also since 2616) provides an ability to send/receive a meta-data out of and after the body part.

HTTP2 also supports it though the chunked TE has been deprecated.

gRPC (https://github.com/grpc/grpc) utilizes it and other applications may also benefit from it.

For XHR2, there was some discussion about trailer support in 2010: https://lists.w3.org/Archives/Public/public-webapps/2010OctDec/thread.html#msg163. @mnot well described that together with observation such as increasing use of the trailers.

Anne declined it at that time https://lists.w3.org/Archives/Public/public-webapps/2010OctDec/0233.html. But maybe it's good to revisit this needs in the context of the Fetch API standardization. Introduction of interfaces to access the trailer may affect streams integration design. So, we should start thinking how it would be like asap if we support it.

dictionary RequestInit {
  ...
  Promise<HeadersInit> trailer;
  ...
};

interface Request {
  ...
  readonly attribute Promise<Headers> trailer;
  ...
};

dictionary ResponseInit {
  ...
  Promise<HeadersInit> trailer;
  ...
};

interface Response {
  ...
  readonly attribute Promise<Headers> trailer;
  ...
};

[annevk]

So trailer headers are atomic just like normal headers are atomic? How are streams affected?

Also, I thought that semantically trailer headers are not distinct from normal headers, though it's not entirely clear to me how that's supposed to work in a streaming situation.

[tyoshino]

Yes, we need to discuss that point. /cc @domenic

Like the headers being atomic, my gut feeling is that the trailers should also be handled without backpressure once the body has been consumed. So, you may need to wait for completion of body consumption by the ReadableStream due to backpressure, or it's ok that a UA fulfills the trailer promise without waiting for completion of consumption if possible.

not distinct

https://tools.ietf.org/html/rfc7230#section-4.1.2 says

   When a chunked message containing a non-empty trailer is received,
   the recipient MAY process the fields (aside from those forbidden
   above) as if they were appended to the message's header section.

Not fully sure, but I interpret this as it doesn't forbid distinction between headers and trailers at the application level.

[annevk]

Given that section 4.1.2 it seems we also need to define what happens when a server does generate those forbidden trailer headers. Will we just pass them through or discard them. And if we discard them, what is our whitelist/blacklist?

I think the UA should fulfill the trailer header promise once the network layer has consumed all trailer header bytes and parsed them successfully. I don't think it needs to depend on content consuming a stream, but I might be missing a subtlety.

[louiscryan]

You should also consider how you want to allow client generated requests to produce trailers. As chunked-encoding must be used in this case and this would have to be specified before the body is produced for HTTP/1.1.

https://tools.ietf.org/html/rfc7230#section-4.1

Separately the client needs to indicate whether it is willing to accept trailers in the response using "TE: trailers", specifying this via the normal request headers mechanism seems sufficient.

https://tools.ietf.org/html/rfc7230#section-4.1.2

Given that trailers are strictly limited to occurring after the last transfer chunk and that the content of the trailers may vary based on the application behavior right up until writing the last chunked byte it seems the API should not require the trailer values prior to receiving the last byte on the stream.

A decision also needs to be made about how to interpret the "Trailer" header itself, though given that it's a SHOULD in the spec I suspect it should just be propagated with the initial header values rather than using it to implement some strict presence checks for the subsequent trailers

https://tools.ietf.org/html/rfc7230#section-4.4

[domenic]

I think streaming should not be too affected. It makes sense to fulfill the trailers promise as soon as they are available---which might be never if the stream consumer exerts backpressure and stops the flow, or might be before the stream consumer fully reads all chunks if the stream is doing internal buffering. And assuming they are used for similar metadata to headers I don't see a need to stream the trailers; we can use the same model as headers and get them all at once.

[annevk]

@louiscryan I think what @tyoshino suggested for requests makes sense. You supply a promise at the point you initiate the request. That indicates ahead of time you might supply trailers.

[louiscryan]

Agreed. Promises seem like a good way to model the intent even if the value
is never used as a result of error.

[annevk]

Another thing we need to define is the relationship between a response having a trailer and the load event for various (non-fetch(), fetch() doesn't do events) features. Does a trailer (that potentially stalls) delay the load event or not? In other words, what's the relationship with process response end-of-file. I guess it would happen after and we'd introduce process response trailer?

[tyoshino]

Separately the client needs to indicate whether it is willing to accept trailers in the response using "TE: trailers", specifying this via the normal request headers mechanism seems sufficient.

It's forbidden to set the TE header via the request headers construction API. Need a method to instruct the Fetch to add it. Maybe a boolean in the RequestInit?

[annevk]

@tyoshino ah yes, for the response we would need that. Perhaps requiring either responseTrailer : "preserve" or supplying a trailer argument with a promise as per your strawman.

[martinthomson]

Note that trailers are very poorly supported in HTTP implementations. Microsoft have explicitly said that they ignore them; others do similar things.

I think that the right thing to do now is to give up on trailers.

[louiscryan]

Not sure I follow your reasoning. Trailers are part of the spec and the historically poor support in server implementations is in no small part because there was no browser API which this proposal would address. Trailers do have support in a non-trivial number of the more popular server libraries and proxies. HTTP2 does a reasonable job addressing the encoding issues that have also dogged trailers.

Is the functional utility of trailers if they were well supported in question? E.g. MACs, trailing status & debug info.

[martinthomson]

If the use cases for trailers were actually compelling, then we would have seen client implementations by now. And it's not just browsers that matter here, you have to convince developers of major client libraries (I'm thinking Windows, iOS and Android here) that it's worthwhile too.

"Build it and they will come" is a posture that ignores the fact that this feature has gone 15 years without any real traction.

[louiscryan]

Last time I looked library support was actually pretty decent

.Net https://msdn.microsoft.com/en-us/library/system.net.http.headers.httprequestheaders.trailer(v=vs.118).aspx
Plain Java (URLConnection, HttpClient, Netty, Jetty & most servlet implementations)
Android Java (OkHttp & URLConnection(?))
libcurl

Ill let others weigh in on iOS

[mnot]

Trailers are used extensively inside of back-end networks and CDNs. Server-side folks generally are very keen to get them supported because they help deal with pain around buffering, etc. E.g., Varnish would LOVE to be able to send ETag in trailers.

The problem has always been browser support. The use cases might not be compelling to browsers, and browsers might have legitimate concerns about buffering, code complexity, etc., but saying that there aren't any compelling use cases because browsers haven't done it yet is pretty presumptuous.