Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Json Source Generator no code generated for Deserialize? #78602

Closed
Alois-xx opened this issue Nov 19, 2022 · 2 comments
Closed

Json Source Generator no code generated for Deserialize? #78602

Alois-xx opened this issue Nov 19, 2022 · 2 comments
Labels
area-System.Text.Json question Answer questions and provide assistance, not an issue with source code or documentation.

Comments

@Alois-xx
Copy link
Contributor

I was trying out source generators with .NET 7.0 and found a nice win of 20+% for the serialize case. Deserialize on the other hand did not really change, except for warmup.

When I set GenerationMode = JsonSourceGenerationMode.Serialization the code fails with an exception during deserialize.

System.InvalidOperationException: TypeInfoResolver 'SerializerTests.Serializers.MyJsonContext' did not provide property metadata for type 'SerializerTests.TypesToSerialize.BookShelf'.
   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_NoMetadataForTypeProperties(IJsonTypeInfoResolver resolver, Type type)
   at System.Text.Json.JsonSerializer.LookupProperty(Object obj, ReadOnlySpan`1 unescapedPropertyName, ReadStack& state, JsonSerializerOptions options, Boolean& useExtensionProperty, Boolean createExtensionProperty)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonTypeInfo jsonTypeInfo)
   at System.Text.Json.JsonSerializer.ReadFromStream[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo)
   at SerializerTests.Serializers.SystemTextJsonSourceGen`1.Deserialize(Stream stream) in D:\Source\git\SerializerTests\Serializers\SystemTextJsonSourceGen.cs:line 75
   at SerializerTests.TestBase`2.TestDeserializeOnlyAndTouch(MemoryStream dataStream, Int32 nExpectedObjects, T& deserialized) in D:\Source\git\SerializerTests\TestBase.cs:line 247
   at SerializerTests.TestBase`2.<>c__DisplayClass39_0.<TestDeserialize>b__0() in D:\Source\git\SerializerTests\TestBase.cs:line 229
   at SerializerTests.TestBase`2.Test(Int32 n, Action acc, Boolean isSerialize) in D:\Source\git\SerializerTests\TestBase.cs:line 168
   at SerializerTests.TestBase`2.TestDeserialize(Int32 nTimes, Int32 nObjectsToCreate) in D:\Source\git\SerializerTests\TestBase.cs:line 226
   at SerializerTests.Test_O_N_Behavior.TestDeserialize(Int32[] nObjectsToDeSerialize, Int32 nRuns) in D:\Source\git\SerializerTests\Test_O_N_Behavior.cs:line 58
   at SerializerTests.Program.Deserialize() in D:\Source\git\SerializerTests\Program.cs:line 557
   at SerializerTests.Program.Run() in D:\Source\git\SerializerTests\Program.cs:line 505
   at SerializerTests.Program.Main(String[] args) in D:\Source\git\SerializerTests\Program.cs:line 404

Does this mean the mode JsonSourceGenerationMode.Serialization only generates code for the serialization logic, but not for deserialization? I this the reason why the default is to generate both? Is there some feature missing to not generate the metadata, but use pregenerated code for serialize and deserialize?
When I can have generated code why should I ever want to enable JsonSourceGenerationMode.Metadata, except to get some fallback when the generated code does not support some scenarios? Will there be a JsonSourceGenerationMode.Deserialization mode, or did I just call the wrong overload to use the pregenerated code?

    //
    // Summary:
    //     The generation mode for the System.Text.Json source generator.
    [Flags]
    public enum JsonSourceGenerationMode
    {
        //
        // Summary:
        //     When specified on System.Text.Json.Serialization.JsonSourceGenerationOptionsAttribute.GenerationMode,
        //     indicates that both type-metadata initialization logic and optimized serialization
        //     logic should be generated for all types. When specified on System.Text.Json.Serialization.JsonSerializableAttribute.GenerationMode,
        //     indicates that the setting on System.Text.Json.Serialization.JsonSourceGenerationOptionsAttribute.GenerationMode
        //     should be used.
        Default = 0,
        //
        // Summary:
        //     Instructs the JSON source generator to generate type-metadata initialization
        //     logic.
        Metadata = 1,
        //
        // Summary:
        //     Instructs the JSON source generator to generate optimized serialization logic.
        Serialization = 2
    }

`

[JsonSourceGenerationOptions(GenerationMode = JsonSourceGenerationMode.Default, IncludeFields = true)]
[JsonSerializable(typeof(BookShelf))]
[JsonSerializable(typeof(BookShelf1))]
[JsonSerializable(typeof(BookShelf2))]
[JsonSerializable(typeof(LargeBookShelf))]
internal partial class MyJsonContext : JsonSerializerContext
{
}

/// <summary>
/// Source Generators for .NET were added with .NET 7.0 which generate de/serialization code during compilation
/// </summary>
/// <typeparam name="T"></typeparam>
[SerializerType("https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/source-generation See docu for Source Generators",
                SerializerTypes.Json | SerializerTypes.SupportsVersioning)]
class SystemTextJsonSourceGen<T> : TestBase<T, JsonSerializerOptions>
{
    public SystemTextJsonSourceGen(Func<int, T> testData, Action<T, int, int> touchAndVerify) : base(testData, touchAndVerify)
    {
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    protected override void Serialize(T obj, Stream stream)
    {
        // Use overload which uses precompiled serialization code
        string dat = JsonSerializer.Serialize(obj, MyJsonContext.Default.Options);

        using IMemoryOwner<byte> utf8_Owner = MemoryPool<byte>.Shared.Rent(dat.Length*2);

        // we still need to convert back to UTF8 or we will write UTF-16 with double size to disk
        int bytes = Encoding.UTF8.GetBytes(dat, utf8_Owner.Memory.Span);
        Span<byte> utf8Span = utf8_Owner.Memory.Span[..bytes];
                    
        stream.Write(utf8Span);
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    protected override T Deserialize(Stream stream)
    {
        // Currently we need to use the MetaData approach for deserialization. 
        // Precompiled Deserialization seems not to be supported yet? 

        if (typeof(T) == typeof(BookShelf))
        {
            return (T) (object) JsonSerializer.Deserialize(stream, MyJsonContext.Default.BookShelf);
        }
        else if (typeof(T) == typeof(BookShelf1))
        {
            return (T) JsonSerializer.Deserialize(stream, MyJsonContext.Default.BookShelf1.Type, MyJsonContext.Default);
        }
        else if (typeof(T) == typeof(BookShelf2))
        {
            return (T) JsonSerializer.Deserialize(stream, MyJsonContext.Default.BookShelf2.Type, MyJsonContext.Default);
        }
        else if (typeof(T) == typeof(LargeBookShelf))
        {
            return (T)JsonSerializer.Deserialize(stream, MyJsonContext.Default.LargeBookShelf.Type, MyJsonContext.Default);
        }

        throw new NotSupportedException($"No source generator for type {typeof(T).Name} declared.");

    }

`

@Alois-xx Alois-xx added the tenet-performance Performance related issue label Nov 19, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 19, 2022
@ghost
Copy link

ghost commented Nov 19, 2022

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

I was trying out source generators with .NET 7.0 and found a nice win of 20+% for the serialize case. Deserialize on the other hand did not really change, except for warmup.

When I set GenerationMode = JsonSourceGenerationMode.Serialization the code fails with an exception during deserialize.

System.InvalidOperationException: TypeInfoResolver 'SerializerTests.Serializers.MyJsonContext' did not provide property metadata for type 'SerializerTests.TypesToSerialize.BookShelf'.
   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_NoMetadataForTypeProperties(IJsonTypeInfoResolver resolver, Type type)
   at System.Text.Json.JsonSerializer.LookupProperty(Object obj, ReadOnlySpan`1 unescapedPropertyName, ReadStack& state, JsonSerializerOptions options, Boolean& useExtensionProperty, Boolean createExtensionProperty)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonTypeInfo jsonTypeInfo)
   at System.Text.Json.JsonSerializer.ReadFromStream[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo)
   at SerializerTests.Serializers.SystemTextJsonSourceGen`1.Deserialize(Stream stream) in D:\Source\git\SerializerTests\Serializers\SystemTextJsonSourceGen.cs:line 75
   at SerializerTests.TestBase`2.TestDeserializeOnlyAndTouch(MemoryStream dataStream, Int32 nExpectedObjects, T& deserialized) in D:\Source\git\SerializerTests\TestBase.cs:line 247
   at SerializerTests.TestBase`2.<>c__DisplayClass39_0.<TestDeserialize>b__0() in D:\Source\git\SerializerTests\TestBase.cs:line 229
   at SerializerTests.TestBase`2.Test(Int32 n, Action acc, Boolean isSerialize) in D:\Source\git\SerializerTests\TestBase.cs:line 168
   at SerializerTests.TestBase`2.TestDeserialize(Int32 nTimes, Int32 nObjectsToCreate) in D:\Source\git\SerializerTests\TestBase.cs:line 226
   at SerializerTests.Test_O_N_Behavior.TestDeserialize(Int32[] nObjectsToDeSerialize, Int32 nRuns) in D:\Source\git\SerializerTests\Test_O_N_Behavior.cs:line 58
   at SerializerTests.Program.Deserialize() in D:\Source\git\SerializerTests\Program.cs:line 557
   at SerializerTests.Program.Run() in D:\Source\git\SerializerTests\Program.cs:line 505
   at SerializerTests.Program.Main(String[] args) in D:\Source\git\SerializerTests\Program.cs:line 404

Does this mean the mode JsonSourceGenerationMode.Serialization only generates code for the serialization logic, but not for deserialization? I this the reason why the default is to generate both? Is there some feature missing to not generate the metadata, but use pregenerated code for serialize and deserialize?
When I can have generated code why should I ever want to enable JsonSourceGenerationMode.Metadata, except to get some fallback when the generated code does not support some scenarios? Will there be a JsonSourceGenerationMode.Deserialization mode, or did I just call the wrong overload to use the pregenerated code?

    //
    // Summary:
    //     The generation mode for the System.Text.Json source generator.
    [Flags]
    public enum JsonSourceGenerationMode
    {
        //
        // Summary:
        //     When specified on System.Text.Json.Serialization.JsonSourceGenerationOptionsAttribute.GenerationMode,
        //     indicates that both type-metadata initialization logic and optimized serialization
        //     logic should be generated for all types. When specified on System.Text.Json.Serialization.JsonSerializableAttribute.GenerationMode,
        //     indicates that the setting on System.Text.Json.Serialization.JsonSourceGenerationOptionsAttribute.GenerationMode
        //     should be used.
        Default = 0,
        //
        // Summary:
        //     Instructs the JSON source generator to generate type-metadata initialization
        //     logic.
        Metadata = 1,
        //
        // Summary:
        //     Instructs the JSON source generator to generate optimized serialization logic.
        Serialization = 2
    }

`

[JsonSourceGenerationOptions(GenerationMode = JsonSourceGenerationMode.Default, IncludeFields = true)]
[JsonSerializable(typeof(BookShelf))]
[JsonSerializable(typeof(BookShelf1))]
[JsonSerializable(typeof(BookShelf2))]
[JsonSerializable(typeof(LargeBookShelf))]
internal partial class MyJsonContext : JsonSerializerContext
{
}

/// <summary>
/// Source Generators for .NET were added with .NET 7.0 which generate de/serialization code during compilation
/// </summary>
/// <typeparam name="T"></typeparam>
[SerializerType("https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/source-generation See docu for Source Generators",
                SerializerTypes.Json | SerializerTypes.SupportsVersioning)]
class SystemTextJsonSourceGen<T> : TestBase<T, JsonSerializerOptions>
{
    public SystemTextJsonSourceGen(Func<int, T> testData, Action<T, int, int> touchAndVerify) : base(testData, touchAndVerify)
    {
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    protected override void Serialize(T obj, Stream stream)
    {
        // Use overload which uses precompiled serialization code
        string dat = JsonSerializer.Serialize(obj, MyJsonContext.Default.Options);

        using IMemoryOwner<byte> utf8_Owner = MemoryPool<byte>.Shared.Rent(dat.Length*2);

        // we still need to convert back to UTF8 or we will write UTF-16 with double size to disk
        int bytes = Encoding.UTF8.GetBytes(dat, utf8_Owner.Memory.Span);
        Span<byte> utf8Span = utf8_Owner.Memory.Span[..bytes];
                    
        stream.Write(utf8Span);
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    protected override T Deserialize(Stream stream)
    {
        // Currently we need to use the MetaData approach for deserialization. 
        // Precompiled Deserialization seems not to be supported yet? 

        if (typeof(T) == typeof(BookShelf))
        {
            return (T) (object) JsonSerializer.Deserialize(stream, MyJsonContext.Default.BookShelf);
        }
        else if (typeof(T) == typeof(BookShelf1))
        {
            return (T) JsonSerializer.Deserialize(stream, MyJsonContext.Default.BookShelf1.Type, MyJsonContext.Default);
        }
        else if (typeof(T) == typeof(BookShelf2))
        {
            return (T) JsonSerializer.Deserialize(stream, MyJsonContext.Default.BookShelf2.Type, MyJsonContext.Default);
        }
        else if (typeof(T) == typeof(LargeBookShelf))
        {
            return (T)JsonSerializer.Deserialize(stream, MyJsonContext.Default.LargeBookShelf.Type, MyJsonContext.Default);
        }

        throw new NotSupportedException($"No source generator for type {typeof(T).Name} declared.");

    }

`

Author: Alois-xx
Assignees: -
Labels:

area-System.Text.Json, tenet-performance

Milestone: -

@eiriktsarpalis
Copy link
Member

Fast-path deserialization has not been implemented yet, see #55043. If you need your source generator to support deserialization you should use the default JsonSourceGenerationMode in your configuration.

@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Nov 21, 2022
@eiriktsarpalis eiriktsarpalis added question Answer questions and provide assistance, not an issue with source code or documentation. and removed tenet-performance Performance related issue labels Nov 21, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Dec 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Text.Json question Answer questions and provide assistance, not an issue with source code or documentation.
Projects
None yet
Development

No branches or pull requests

2 participants