Repository navigation
Developers should be able to pass state to custom converters. #63795
Description
Background and Motivation
The current model for authoring custom converters in System.Text.Json is general-purpose and powerful enough to address most serialization customization requirements. Where it falls short currently is in the ability to accept user-provided state scoped to the current serialization operation; this effectively blocks a few relatively common scenaria:
-
Custom converters requiring dependency injection scoped to the current serialization. A lack of a reliable state passing mechanism can prompt users to rebuild the converter cache every time a serialization operation is performed.
-
Custom converters do not currently support streaming serialization. Built-in converters avail of the internal "resumable converter" abstraction, a pattern which allows partial serialization and deserialization by marshalling the serialization state/stack into a state object that gets passed along converters. It lets converters suspend and resume serialization as soon as the need to flush the buffer or read more data arises. This pattern is implemented using the internal
JsonConveter<T>.TryWriteandJsonConverter<T>.TryReadmethods.Since resumable converters are an internal implementation detail, custom converters cannot support resumable serialization. This can create performance problems in both serialization and deserialization:
- In async serialization, System.Text.Json will delay flushing the buffer until the custom converter (and any child converters) have completed writing.
- In async deserialization, System.Text.Json will need to read ahead all JSON data at the current depth to ensure that the custom converter has access to all required data at the first read attempt.
We should consider exposing a variant of that abstraction to advanced custom converter authors.
-
Custom converters are not capable of passing internal serialization state, often resulting in functional bugs when custom converters are encountered in an object graph (cf. ReferenceHandler.IgnoreCycles doesn't work with Custom Converters #51715, Incorrect JsonException.Path when using non-leaf custom JsonConverters #67403, Keep LineNumber, BytePositionInLine and Path when calling JsonSerializer.Deserialize<TValue>(ref reader) #77345)
Proposal
Here is a rough sketch of how this API could look like:
namespace System.Text.Json.Serialization;
public struct JsonWriteState
{
public CancellationToken { get; }
public Dictionary<string, object> UserState { get; }
}
public struct JsonReadState
{
public CancellationToken { get; }
public Dictionary<string, object> UserState { get; }
}
public abstract class JsonStatefulConverter<T> : JsonConverter<T>
{
public abstract void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options, ref JsonWriteState state);
public abstract T? Read(ref Utf8JsonReader writer, Type typeToConvert, JsonSerializerOptions options, ref JsonReadState state);
// Override the base methods: implemented in terms of the new virtuals and marked as sealed
public sealed override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options) {}
public sealed override T? Read(ref Utf8JsonReader writer, Type typeToConvert, JsonSerializerOptions options) {}
}
public abstract class JsonResumableConverter<T> : JsonConverter<T>
{
public abstract bool TryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, ref JsonWriteState state);
public abstract bool TryRead(ref Utf8JsonReader writer, Type typeToConvert, JsonSerializerOptions options, ref JsonReadState state, out T? result);
// Override the base methods: implemented in terms of the new virtuals and marked as sealed
public sealed override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options) {}
public sealed override T? Read(ref Utf8JsonReader writer, Type typeToConvert, JsonSerializerOptions options) {}
}
public partial class JsonSerializer
{
// Overloads to existing methods accepting state
public static string Serialize<T>(T value, JsonSerializerOptions options, ref JsonWriteState state);
public static string Serialize<T>(T value, JsonTypeInfo<T> typeInfo, ref JsonWriteState state);
public static T? Deserialize<T>(string json, JsonSerializerOptions options, ref JsonReadState state);
public static T? Deserialize<T>(string json, JsonTypeInfo<T> typeInfo, ref JsonReadState state);
// New method groups enabling low-level streaming serialization
public static bool TrySerialize(T value, JsonTypeInfo<T> typeInfo, ref JsonWriteState state);
public static bool TryDeserialize(string json, JsonTypeInfo<T> typeInfo, ref JsonReadState state);
}Users should be able to author custom converters that can take full advantage of async serialization, and compose correctly with the contextual serialization state. This is particularly important in the case of library authors, who might want to extend async serialization support for custom sets of types. It could also be used to author top-level async serialization methods that target other data sources (e.g. using System.IO.Pipelines cf. #29902)
Usage Examples
MyPoco value = new() { Value = "value" };
JsonWriteState state = new() { UserState = { ["sessionId"] = "myId" }};
JsonSerializer.Serialize(value, options, state); // { "sessionId" : "myId", "value" : "value" }
public class MyConverter : JsonStatefulConverter<MyPoco>
{
public override void Write(Utf8JsonWriter writer, MyPoco value, JsonSerializerOptions options, ref JsonWriteState state)
{
writer.WriteStartObject();
writer.WriteString("sessionId", (string)state.UserState["sessionId"]);
writer.WriteString("value", value.Value);
}
}Alternative designs
We might want to consider the viability attaching the state values as properties on Utf8JsonWriter and Utf8JsonReader. It would avoid the need of introducing certain overloads, but on the flip side it could break scenaria where the writer/reader objects are being passed to nested serialization operations.
Goals
- Support custom resumable converters.
- Support custom converters that are passing the serialization state to child converters.
- Support async serialization using data sources other than
Stream(à la [API Proposal]: JsonSerializer.TryReadValue(ref Utf8JsonReader) #29902). - Support users attaching custom state to serialization operations ([API Proposal]: Support Custom Data on
JsonSerializerOptions#71718)
Progress
- Author prototype
- API proposal & review
- Implementation & tests
- Conceptual documentation & blog posts.
Activity
eiriktsarpalis commentedon Jan 14, 2022
Tagging @steveharter who might be interested in this.
jeffhandley commentedon Apr 6, 2022
I'm updating this feature's milestone to Future as it is not likely to make it into .NET 7.
19 remaining items
eiriktsarpalis commentedon May 31, 2024
The types already exist as internal implementation detail and hold additional state which wouldn't be visible to users.
JustDre commentedon Nov 11, 2024
Given that #64182 was closed in deference to this issue, I don't know why the solution needs to involve async converters when DeserializeAsyncEnumerable does not and yet provides significant memory savings over DeserializeAsync (whenever streaming deserialization is appropriate). It seems the ability to call DeserializeAsyncEnumerable at any arbitrary point of an incoming stream, and leaving the stream open at the appropriate byte location upon exit, would be immensely useful. While the solutions discussed above are optimal from a resource perspective, forcing all developers to fully support async converters seems like a bigger lift than necessary, simply to support non-root array deserialization. If an overload can be implemented to achieve this, then once async converters become available, developers can opt in to implement them as appropriate.
JustDre commentedon Nov 11, 2024
Maybe it's not important, but issue #77018 (that tracks this one) is currently closed. I don't see it being tracked by an equivalent issue for .NET 9, but since it's days (hours?) away from release, it probably needs to be tracked by the equivalent for .NET 10.
Harpush commentedon Apr 3, 2025
Any chance for non root deserialize to IAsyncEnumerable? Many APIs return the array inside an object thus eliminating the possibility to use IAsyncEnumerable
eiriktsarpalis commentedon Apr 4, 2025
How would this work? Assuming you were trying to stream a large JSON object like so
{ "prop1" : 1, "largeProp" : [1,2,3,4,5,....], "prop2": 2 }into the type
the returned value should have been hydrated with data for all its properties. How could this be reconciled with streaming?
airbreather commentedon Apr 4, 2025
I'm guessing the idea (because I have a similar use case to this) would be that we have a root object that is essentially nothing more than a container for an array? GeoJSON FeatureCollection comes to mind.
Edit to elaborate: the implication being that we would deserialize only the inner array for a given property as though it were top-level, and all other properties would be ignored (except maybe for the purposes of raising format errors or something).
IMO not worth significant investment because it only works for narrow use cases, but not fundamentally incompatible or useless either.
eiriktsarpalis commentedon Apr 4, 2025
That is definitely achievable. The potential shape we've been thinking about is a
JsonSerializer.DeserializeAsyncEnumerableoverload that accepts a JSON pointer to the nested location that needs to be streamed. It means that any other contextual information would be discarded.Harpush commentedon Apr 4, 2025
Discarding is the way to go. Let's put aside streaming non array jsons as I am really not sure there is any way to do it except for tokens.
The idea is for cases like
{ "data": [/* huge array */] }. I have zero interest in data wrapper and I would be happy with the service returning just an array. So ideally I would expect a way to specify path or even property where the array exists.Another more low level oriented approach can be to allow getting a tokens stream from http stream skipping whatever you want and then an option to say deserialize async enumerable from here.
This will allow to easily even parse (as one big concatenated array) a json like
{data1: [], data2: []}.eiriktsarpalis commentedon Apr 4, 2025
Makes sense. I would say though that this is out of scope for this particular issue. Could you create a new one please?
Harpush commentedon Apr 4, 2025
#114265
thomasnalix commentedon Nov 27, 2025
Any update on this?
This has been pushed back across several releases, so it’s hard to understand what’s still blocking it or whether there’s a chance for .NET 11
eiriktsarpalis commentedon Nov 27, 2025
It's a resourcing issue.
NWalker1208 commentedon Sep 30, 2026
It's been over four years since this issue was opened, and I'm a bit disappointed that it hasn't seen much progress since then. The fact that state can't be propagated through custom converters has been a consistent source of frustration in the projects I've worked on. Specifically, the fact that a custom converter deserializing a JSON object truncates the JSON paths of any exceptions that occur when deserializing values deeper within the object (see #67403). I've had to resort to various workarounds as a result. It would be great to see this aspect of System.Text.Json improved.
bkoelman commentedon Oct 1, 2026
@NWalker1208 Totally agree. This is the biggest limitation in STJ, as far as I'm concerned.
andreascaesar commentedon Oct 1, 2026
We are working a lot with STJ and custom converters, and this is really the only thing we are missing in STJ...
So, by far the biggest limitation!