Avro inferred mapping issues with nested struct

[jzhuge]

When nameMapping is null, projected schema is automatically inferred from read schema. We discovered issues when there is extra nested struct.

1. When there are two extra nested structs: projected schema can not be serialized to json because 2 types in the nested struct have the same name rnull
2. When there is one extra nested struct: json serialization is fine but schema type name rnull still does not seem right

Unit tests:

  @Test
  public void testInferredMappingNestedStruct() throws IOException {
    Schema writeSchema = new Schema(
        Types.NestedField.required(0, "id", Types.LongType.get())
    );

    Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
    record.put("id", 34L);

    Schema readSchema = new Schema(
        Types.NestedField.required(0, "id", Types.LongType.get()),
        Types.NestedField.optional(3, "location", Types.StructType.of(
            Types.NestedField.required(1, "lat", Types.FloatType.get()),
            Types.NestedField.required(2, "long", Types.FloatType.get())))
    );

    Record projected = writeAndRead(writeSchema, readSchema, record, null);
    AvroTestHelpers.assertEquals(writeSchema.asStruct(), record, projected);

    validateJsonSerialization(projected);
    validateNewNestedStruct(projected, readSchema.findField("location"));
  }

  @Test
  public void testInferredMappingTwoNestedStructs() throws IOException {
    Schema writeSchema = new Schema(
        Types.NestedField.required(0, "id", Types.LongType.get())
    );

    Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
    record.put("id", 34L);

    Schema readSchema = new Schema(
        Types.NestedField.required(0, "id", Types.LongType.get()),
        Types.NestedField.optional(3, "location", Types.StructType.of(
            Types.NestedField.required(1, "lat", Types.FloatType.get()),
            Types.NestedField.required(2, "long", Types.FloatType.get()))),
        Types.NestedField.optional(4, "address", Types.StructType.of(
            Types.NestedField.required(5, "street", Types.IntegerType.get())))
    );

    Record projected = writeAndRead(writeSchema, readSchema, record, null);
    AvroTestHelpers.assertEquals(writeSchema.asStruct(), record, projected);

    validateJsonSerialization(projected);
    validateNewNestedStruct(projected, readSchema.findField("location"));
    validateNewNestedStruct(projected, readSchema.findField("address"));
  }

  private void validateJsonSerialization(Record projected) {
    Assert.assertFalse("Projected schema can be serialized to json", projected.getSchema().toString().isEmpty());
  }

  private void validateNewNestedStruct(Record projected, Types.NestedField field) {
    String newTypeName = "r" + field.fieldId();
    String newFieldName = field.name() + "_" + newTypeName;
    Assert.assertNull(field.name() + " field should be null", projected.get(newFieldName));
    Assert.assertNotNull(field.name() + " field is renamed to " + newFieldName,
        projected.getSchema().getField(newFieldName));
    Assert.assertEquals(field.name() + " field should have schema type name " + newTypeName,
        newTypeName, projected.getSchema().getField(newFieldName).schema().getTypes().get(1).getName());
  }

[jzhuge]

@rdblue @rdsr Could you take a look?

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'