Custom JsonConverter<string> implementation throws NotSupportedException when serializing a dictionary of strings in .NET 6 preview 4

[martincostello]

Description

In a line-of-business application we have the following custom JsonConverter for strings used to remove "bad" Unicode from strings in serialized JSON.

internal sealed class BadUnicodeRemovingJsonConverter : JsonConverter<string>
{
    public override string Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        => reader.GetString();

    public override void Write(Utf8JsonWriter writer, string value, JsonSerializerOptions options)
    {
        static string RemoveBadUnicode(string input, string pattern)
        {
            return Regex.Replace(input, pattern, string.Empty, RegexOptions.IgnoreCase);
        }

        string sanitized = RemoveBadUnicode(value, "[\u0000-\u0009]");
        sanitized = RemoveBadUnicode(sanitized, "[\u000b-\u000c]");
        sanitized = RemoveBadUnicode(sanitized, "[\u000e-\u001f]");

        writer.WriteStringValue(sanitized);
    }
}

After updating the application to .NET 6 preview 4, a NotSupportedException is thrown with a message similar to:

System.NotSupportedException: The type 'System.String' is not a supported dictionary key using converter of type 'MyApp.BadUnicodeRemovingJsonConverter'. Path: $.Foo.Bar.

For the specific type being serialized, $.Foo.Bar is an IDictionary<string, string> property Bar on a custom type Foo, which is in turn a property of another custom type at the root.

public class RootObject
{
    public ChildObject Foo { get; set; }
}

public class ChildObject
{
    public IDictionary<string, string> Bar { get; set; }
}

The object is serialized with the following settings:

var options = new JsonSerializerOptions
{
    WriteIndented = false,
};

options.Converters.Add(new BadUnicodeRemovingJsonConverter());

string json = JsonSerializer.Serialize(rootObject, options);

Configuration

.NET 6 preview 4 (SDK version 6.0.100-preview.4.21255.9)

Regression?

Yes, this code works today with .NET 5.0.6. The issue was uncovered by the application's tests when updated to .NET 6.0 preview 4. The issue was also not present in previous previews of .NET 6.

Other information

    System.NotSupportedException: The type 'System.String' is not a supported dictionary key using converter of type 'MyApp.BadUnicodeRemovingJsonConverter'. Path: $.Foo.Bar.
     ---> System.NotSupportedException: The type 'System.String' is not a supported dictionary key using converter of type 'MyApp.BadUnicodeRemovingJsonConverter'.
       at System.Text.Json.ThrowHelper.ThrowNotSupportedException_DictionaryKeyTypeNotSupported(Type keyType, JsonConverter converter) in System.Text.Json.dll:token 0x60000f2+0x16
       at System.Text.Json.Serialization.JsonConverter`1.WriteWithQuotes(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006fd+0x0
       at System.Text.Json.Serialization.Converters.IDictionaryOfTKeyTValueConverter`3.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x6000868+0xc3
       at System.Text.Json.Serialization.Converters.DictionaryDefaultConverter`3.OnTryWrite(Utf8JsonWriter writer, TCollection dictionary, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x6000840+0x70
       at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006f6+0x206
       at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) in System.Text.Json.dll:token 0x60007b7+0x130
       at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60008d8+0xa7
       at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006f6+0x206
       at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) in System.Text.Json.dll:token 0x60007b7+0x130
       at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60008d8+0xa7
       at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006f6+0x206
       at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006e1+0x0
       --- End of inner exception stack trace ---

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

In a line-of-business application we have the following custom JsonConverter for strings used to remove "bad" Unicode from strings in serialized JSON.

internal sealed class BadUnicodeRemovingJsonConverter : JsonConverter<string>
{
    public override string Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        => reader.GetString();

    public override void Write(Utf8JsonWriter writer, string value, JsonSerializerOptions options)
    {
        static string RemoveBadUnicode(string input, string pattern)
        {
            return Regex.Replace(input, pattern, string.Empty, RegexOptions.IgnoreCase);
        }

        string sanitized = RemoveBadUnicode(value, "[\u0000-\u0009]");
        sanitized = RemoveBadUnicode(sanitized, "[\u000b-\u000c]");
        sanitized = RemoveBadUnicode(sanitized, "[\u000e-\u001f]");

        writer.WriteStringValue(sanitized);
    }
}

After updating the application to .NET 6 preview 4, a NotSupportedException is thrown with a message similar to:

System.NotSupportedException: The type 'System.String' is not a supported dictionary key using converter of type 'MyApp.BadUnicodeRemovingJsonConverter'. Path: $.Foo.Bar.

For the specific type being serialized, $.Foo.Bar is an IDictionary<string, string> property Bar on a custom type Foo, which is in turn a property of another custom type at the root.

public class RootObject
{
    public ChildObject Foo { get; set; }
}

public class ChildObject
{
    public IDictionary<string, string> Bar { get; set; }
}

The object is serialized with the following settings:

var options = new JsonSerializerOptions
{
    WriteIndented = false,
};

options.Converters.Add(new BadUnicodeRemovingJsonConverter());

string json = JsonSerializer.Serialize(rootObject, options);

Configuration

.NET 6 preview 4 (SDK version 6.0.100-preview.4.21255.9)

Regression?

Yes, this code works today with .NET 5.0.6. The issue was uncovered by the application's tests when updated to .NET 6.0 preview 4. The issue was also not present in previous previews of .NET 6.

Other information

    System.NotSupportedException: The type 'System.String' is not a supported dictionary key using converter of type 'MyApp.BadUnicodeRemovingJsonConverter'. Path: $.Foo.Bar.
     ---> System.NotSupportedException: The type 'System.String' is not a supported dictionary key using converter of type 'MyApp.BadUnicodeRemovingJsonConverter'.
       at System.Text.Json.ThrowHelper.ThrowNotSupportedException_DictionaryKeyTypeNotSupported(Type keyType, JsonConverter converter) in System.Text.Json.dll:token 0x60000f2+0x16
       at System.Text.Json.Serialization.JsonConverter`1.WriteWithQuotes(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006fd+0x0
       at System.Text.Json.Serialization.Converters.IDictionaryOfTKeyTValueConverter`3.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x6000868+0xc3
       at System.Text.Json.Serialization.Converters.DictionaryDefaultConverter`3.OnTryWrite(Utf8JsonWriter writer, TCollection dictionary, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x6000840+0x70
       at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006f6+0x206
       at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) in System.Text.Json.dll:token 0x60007b7+0x130
       at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60008d8+0xa7
       at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006f6+0x206
       at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) in System.Text.Json.dll:token 0x60007b7+0x130
       at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60008d8+0xa7
       at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006f6+0x206
       at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) in System.Text.Json.dll:token 0x60006e1+0x0
       --- End of inner exception stack trace ---

Author:	martincostello
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[martincostello]

FYI this still repros in 6.0.100-preview.5.21302.13.

[eiriktsarpalis]

Can confirm this is a regression from .NET 5. Minimal reproduction:

using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Text.Json.Serialization;

namespace ConsoleApp1
{
    class Program
    {
        internal sealed class CustomStringConverter : JsonConverter<string>
        {
            public override string? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
                => reader.GetString();

            public override void Write(Utf8JsonWriter writer, string value, JsonSerializerOptions options)
                => writer.WriteStringValue(value);
        }

        static void Main()
        {
            var options = new JsonSerializerOptions
            {
                Converters = { new CustomStringConverter() }
            };

            var value = new Dictionary<string, string>()
            {
                ["key"] = "value"
            };

            JsonSerializer.Serialize(value, options);
        }
    }
}

[eiriktsarpalis]

cc @steveharter @layomia

[root6819]

@martincostello martincostello I met the question too, how do you fix it at last?

[martincostello]

@Byerror I haven't fixed or worked around it - the 3 apps I have that hit this issue just have various broken tests awaiting a fix in a future preview.

[eiriktsarpalis]

On further investigation it looks like this breaking change was made intentionally in #50074 (comment). @layomia should we close this in favor of #46520 or reconsider the behavior that we ship in .NET 6?