Repository navigation
GCP: ClientOptions default_headers (Accept-Encoding) break OAuth token fetch #783
Description
Describe the bug
ClientOptions::client() applies user default_headers (src/client/mod.rs:843) and also sets .no_gzip()... (mod.rs:929). The GCP token providers reuse that same client (src/gcp/builder.rs:606-608). So a data-plane Accept-Encoding: gzip header leaks onto the OAuth token POST (src/gcp/credential.rs:613-637, accounts.google.com/o/oauth2/token); Google returns gzip, no_gzip skips decoding, and .json() fails → Error::TokenResponseBody. The client tells the server "send gzip" while telling reqwest "don't decode gzip".
To Reproduce
let mut h = http::HeaderMap::new();
h.insert("accept-encoding", "gzip".parse().unwrap());
let store = GoogleCloudStorageBuilder::from_env()
.with_bucket_name("my-bucket")
.with_client_options(ClientOptions::new().with_default_headers(h))
.build()?;
store.get(&"x".into()).await?; // authorized_user ADC: token fetch fails before the GETError: Generic { store: "GCS", source: TokenResponseBody(... "expected value at line 1 column 1") }.
(Accept-Encoding: gzip is legitimate: it makes GCS serve Content-Encoding: gzip objects as raw gzip.)
Expected behavior
Setting default_headers for the data plane should not break authentication.
Additional context
Activity
ByteBaker commentedon Jul 6, 2026
I'd argue that this is not wrong. As
ClientOptionshas no awareness of the impact caused by headers passed by the caller. As such, the string "gzip" does not meaning anything to the client, it's just a string, and hence it won't try and derive meanings from the payload of default headers, even if one of those is "gzip".However, the right way IMO would be to track encoding inside
ClientOptions, and turn encoding on/off inside the client builder accordingly. Since the culprit linebuilder = builder.no_gzip().no_brotli().no_zstd().no_deflate();turns off all forms of encoding.@tustvold @alamb thoughts?
ByteBaker commentedon Jul 6, 2026
Upon looking further, it turns out the surface area is far bigger than anticipated.
reqwestdefines the methodsClientBuilder::no_{gzip|brotli|zstd|deflate()like the following.This means that the said feature must be enabled for reqwest to process the given encoding.
However,
object_storedoes not yet provide a way to opt into those reqwest-specific features when usingobject_storeas a dependency. To ensure that a user externally enabling those features in reqwest via a parallel dependency does not change reqwest's behavior, it chooses to opt-out of all those algorithms. This is declared in the comment preceding the code:Having said that, this is clear that object-store intentionally disables all forms of response compression. There's 3 ways forward to this:
Accept-Encodingheader since object store doesn't support it. @nkemnitz's concern is valid, but the contract between the headers and compression features is not enforced by them. (Easiest, but errors can happen if users don't read it)Accept-Encodingheader while storing default headers for the client and declare the behaviour in docs. This still stays with no compression, but the users know that including this header in default headers has no effect, and the client still requests content w/o compression. (Easy, ensures errors don't happen)Content-Lengthproblem mentioned in the comment. (Complex, complete solution to the problem)Please suggest a way forward.
nkemnitz commentedon Jul 6, 2026
To be clear, I need to send
Accept-Encoding: gzipheaders on the actual requests to GCS, otherwise Google will assume my client is unable to handle the gzip compressed objects, decode and serve them via decompressive transcoding instead. That's potentially slow and expensive (uncompressed transfer), and worse: it strips theContent-Lengthheader, which breaks another object-stores assumption - see #774ByteBaker commentedon Jul 8, 2026
@nkemnitz I went through #774, and that seems to be linked to this. Unfortunately 1 & 2 as suggested above won't do you any good. We'll have to see what the team thinks.
alamb commentedon Jul 8, 2026
The latest object store release allows you to use your own client (rather than the built in reqwest) -- maybe that mechanism can be used to configure the headers ?