Add support for Embedded Objects

[cmelchior]

Closes #6713

This PR adds support for "Embedded Objects". For non-synced Realms, the only benefit is that it can be be used to implement cascading deletes in some scenarios. For synced Realms it also describes how the data is serialized once it reaches a MongoDB server.

Semantics

Embedded objects have slightly different semantics than normal objects:

• When the parent of an embedded object is deleted, so is the embedded object.
• Setting the link to the embedded object to null will also delete the embedded object.
• Only one parent can ever link to an embedded object. Embedded objects can be parents to other embedded objects themselves.
• @PrimaryKey is not supported on embedded objects.
• Free-floating embedded objects with no parents are not possible, not even as a temporary state. This also prevents cycles between embedded objects. Only tree-structures are possible.
• The above restrictions also mean it effectively isn't possible to move objects once assigned a parent.

API

// Mark class as embedded
@RealmClass(embedded = true)
public class Person extends RealmObject {

  // @LinkingObjects pointing to a single parent is supported for embedded objects
  // They can optionally be marked @Required, although this is assumed if only one 
  // parent type exists
  @Required
  @LinkingObjects("child")
  public final Parent parent;

  // It is still possible to have multiple different parent types, but only one can be set.
  // This also means that if multiple possible parent types are possible. Neither of the
  // @LinkingObjects can be marked as @Required.
  @LinkingObjects("child1")
  public final Parent parent;
  @LinkingObjects("child2")
  public final Parent parent;

  // The old way of setting @LinkingObjects is still possible. But the result will only ever
  // contain 1 element. 
  @LinkingObjects("child")
  public final RealmResults<Parent> parents = null;
}

// RealmObjectSchema exposes the information
boolean RealmObjectSchema.isEmbedded();

// Changing the embeddedness is only possible if all variants are true
// when changing the flag.
RealmObjectSchema.setEmbedded(boolean);

// Queries on lists are allowed
RealmResults<Person> results = parent.children.where()...findAll();

// Creating embedded objects need to provide their parent as well.
// All `copyToRealm` and `insert` methods do this already
realm.insert(new Parent(new Person()));
realm.insertOrUpdate(new Parent(new Person()));
realm.copyToRealm(new Parent(new Person()));
realm.copyToRealmOrUpdate(new Parent(new Person()));

// Creating objects directly must also provide the linking parent.
// So a new method has been added. If the parent is a list, the 
// semantics implemented is that the object is added to the 
// end of the list. This seemed the most sensible bit is also very subjective.
realm.createEmbeddedObject(Person.class, managedParent, "parentFieldName"); 

// Adding unmanaged objects to either lists or properties will automatically copy them correctly and make them embedded
managedObject.field = EmbeddedPerson() // Will automatically copy the object as embedded 
managedObject.list.add(EmbeddedPerson()) // Will automatically copy the object as embedded 

// Dynamic Realms poses a bit of a challenge, since we don't have a concept of 
// unmanaged DynamicRealmObjects. This currently means that some things do not work.

// Same as for typed Realms
dynamicRealm.createEmbeddedObjcet("parentClass", parentDynamicObject, "parentField");

// The convience methods available to typed Realms are not available for DynamicRealm until we have support for an unmanaged RealmObject. All the the below would does not work for now.
RealmList<DynamicRealmObject> list = obj.getList("myList");
list.add(obj)
list.add(index, obj) 
list.set(index, obj)
obj.setObject(embeddedObject);

// JSON support has currently not been added, but all the normal methods should be made to work

// RealmResults.setObject(...) support has not been added yet. Also unclear exactly how this should work.

TODO

• Upgrade to Sync-alpha.15 to fix last to unit test failures.
• Add annotation processor support + test
• Add tests for support schema graph types: Simple, Tree, Circular
• Extend RealmObjectSchema + tests
• How should migrations for embedded classes be handled? Not relevant for synchronized Realms I guess, but could be relevant for non-synced Realms.
• Tests for copyToRealm
• Tests for insert

TODO for other PR's

• Runtime tests for the possible @LinkingObject configs. Especially in Kotlin.
• Tests for insertandinsertOrUpdate` with lists. All these variants use different compiled code.
• Add support for an unmanaged DynamicRealmObject
• Implementation and tests for createFromJson-variants.
• Figure out exactly how to support RealmResults.setObject(...) + Tests
• Add test involving synchronization of embedded objects
• Fix createEmbeddedObject not checking if the parent object has the correct type
• Fix support for using embedded Model classes defined in another library module.
• copyToRealm and insert are implemented using different strategies. The one used by insert seems much simpler, so would probably be helpful to refactor how copyToRealm is implemented.

[tgoyne]

Any thoughts on calling the one link version LinkingObject (with no s)?

[cmelchior]

Yes, I did consider it. Arguably it would be more correct, but it does introduce one more concept to learn and it makes it more difficult for us to document how to use backlinks in general. Also, the names would be so close together that I could see some having doubts about which one to use or hit the wrong one by accident when autocompleting.

Also, since the syntax is basically just a shorthand for calling parents.first() on a backlink Results, I think just re-using the current annotation is fine.

It was sort of the same reasoning for choosing @RealmClass(embedded = true) instead of a new annotation like @Embedded

[Zhuinden]

the the

[Zhuinden]

That's already there 3 lines above I think?

[cmelchior]

Yup, there are some merge mistakes that needs to be cleaned up

[rorbech]

I gave it a try. Quite a big lump with lots of areas I have not touched yet. Besides the comments, maybe put a note on the 'Dynamic realm constraints' in some more obvious place.

[rorbech]

Maybe also add the generated proxy for the EmbeddedClass for reference.

[edualonso]

Looks massive but overall good. I tried not to get too caught into the business details, so mostly nitpicking from my part.

[edualonso]

There seems to be a lot immutable data coming out of classType. I'm wondering if we could somehow avoid using vars for things like this and rely on initialisation for assigning values instead. It feels like we are adding state to this class unnecessarily. Another example from this class is the contains* booleans.

[cmelchior]

This is probably an architectural problem. Right now ClassMetaData is responsible for both parsing AND storing state. It would probably help if we could separate this a bit. The issue is that there is a lot of state and it is interdependent, so not sure how much it would help if we split it. We would still need to track a lot of intermediate state somewhere.

[edualonso]

It feels like it could escalate quickly if these changes are done here and now. I don't think it's super urgent so we can push it for a later time. I'm very wary when it comes to using mutable properties in Kotlin nonetheless. If not managed correctly they become a timebomb.

[cmelchior]

If you look at the rest of this class there are tons of var as well. That doesn't mean your point is invalid, but I wonder how you would handle the case of gathering ~10+ variables across 100's of line of code before you send them to a constructor?

[edualonso]

I can see e.g. internalClassName is a latent var. It takes its value from realmClassAnnotation or moduleClassNameFormatter (it's a bit more cumbersome to reason in case the value comes from the latter, but still, it's the same rationale as it could be passed via constructor/a builder from where the class metadata is created in RealmProcessor). In turn, the local realmClassAnnotation variable is immutable and takes its value from classType which comes from the constructor as immutable too. All these things considered, there is no excuse to make internalClassName a mutable property.

It is true that you might end up having a gazillion properties in your constructor, to which I would strongly object too. You could use a builder to handle that, but that adds a ton of boilerplate. I think the best solution is to encapsulate all these immutable properties behind a data structure/class. This way you would at least keep the clutter away from ClassMetaData and those properties will be easily accessible via vals with backing properties from ClassMetaData, e.g.

class ClassMetaData(private val myMetadaContainer: MetaDataContainer) {
   private val internalClassName: String
      get() = myMetadaContainer.internalClassName

I don't think there's a silver bullet here in any case but makes the solution more robust IMO.

[cmelchior]

I have no doubt there are things we can do to improve this class. You mentioned some of them, but I would rather not do it in this PR since it is too big already. I created #6891 to track it.

[edualonso]

I agree, seems too much for the scope of this PR.

[edualonso]

I know it's a test, but avoid var unless you're updating the value of children with a different list instance - remember you can still add elements to the list even if its property is immutable.

[cmelchior]

We don't support final fields in the annotation processor. Probably mostly an oversight which is a lot more annoying in Kotlin. I created #6892. I suspect it would be a fairly trivial fix.

[edualonso]

Good to know, I had no idea.

[edualonso]

Does it make sense to have an individual leafNode while having a leafNodeList? Also, no var for the lists.

[cmelchior]

I have both because it is different code paths in the generated code. I agree from a modeling perspective it looks weird.

For the var see above.

[edualonso]

Got it 👍

[savelyevdm]

Hi, guys!
Nice improvement!
I failed to find documentation for migration from ordinary model class to embedded model class.

Lets say, that I have parent class and child class, which connects to parent object by id:

public class MyParent extends RealmObject {
    @PrimaryKey
    private String id;
    private MyChild child; 
    // other fields...
}
public class MyChild extends RealmObject {
    @PrimaryKey
    private String id;
    private String parentId;
    // other fields...
}

Experimentally I found, that I should:

• either (1a) guarantee that every embedded-candidate object in MyChild's table has single parent or (1b) clear model tables
• then (2) remove private key status from private field ("id") & remove field "id" itself from MyChild
• (3) set embedded status to MyChild class.
  In migration code it looks as follows:

            realm.delete("MyParent");
            realm.delete("MyChild"); // optional; variant (1b) is chosen

            schema.get("MyChild")
                    .removePrimaryKey()
                    .removeField("id");

            schema.get("MyChild").setEmbedded(true);

Am I right?

[ianpward]

@savelyevdm Thanks for trying out this new feature! Yes you've discovered the caveats, the embedded object can only have one parent and no primary key.

For Migrations, In Java I believe you can just call RealmObjectSchema.setEmbedded(bool) as part of the migration. setEmbedded() checks the constrains of no primary keys and no more than one parent

Let us know how it works for you

[savelyevdm]

@savelyevdm Thanks for trying out this new feature! Yes you've discovered the caveats, the embedded object can only have one parent and no primary key.

For Migrations, In Java I believe you can just call RealmObjectSchema.setEmbedded(bool) as part of the migration. setEmbedded() checks the constrains of no primary keys and no more than one parent

Let us know how it works for you

Thanx for the fast reply!

As my MyChild class already had primary key - so it wasn't possible not to delete the primary key & this field. Hope I've done it correctly?

[ianpward]

@savelyevdm Gotcha - is no primary key a blocker for you? Do you need a primary key for this object?

[GZaccaroni]

Hi guys!
Is it possible to add an embedded object while migrating?
I've tried with:

let embeddedObj = MigrationObject()
embeddedObj["value"] = "10"
embeddedObj["trend"] = 1
parentObj["emb"] = embeddedObj

and:

let embeddedObj = migration.create(EmbeddedObj.className())
embeddedObj["value"] = "10"
embeddedObj["trend"] = 1
parentObj["emb"] = embeddedObj

but both attempts failed.

Thank you!

[cmelchior]

Yes, you can create embedded objects during an migration, but you need to set the embedded flag on the class RealmObjectSchema.setEmbedded(true)

If that doesn't work. Can you create a new issue with the code you tried so far?

[sipersso]

Is there any guidance in the docs on how to migrate a collection to use embedded objects? Spent an hour looking through the documentation and could not find any guidance. The RealmObjectSchema class mentioned here does not seem to exist in the Swift library? I need to do this migration on both iOS and Android in order to be able to start syncing to MongoDb Realm

[sipersso]

@cmelchior What is the corresponding function for RealmObjectSchema.setEmbedded(true) in swift. I am trying to migrate my old data model from regular objects to embedded objects. Is this possible?