Consume monorepo core binaries #7241

clementetb · 2020-12-09T17:37:40Z

This PR adds the option to consume prebuild realm-core monorepo release artifacts, with the intention of minimizing the CI timings.

Allow to consume realm-core release artifacts.
Update Jenkins file to consume binaries instead.
Update Readme with new build instructions.

Warning: This PR will fail to run due to an error in the CMake file distributed in the release artifacts.

* Use monorepo * Use newer CMake * use latest monorepo * Point to right cmake version Jenkins to use right cmake version Fix imports and executables linking * Fix cmake url * Add ninja-build package to docker image * Add new cmake instructions to readme * don’t worry about realm-core’s command-line targets * use latest monorepo * Add support for LTO. It is enabled by default for release builds, but can be opted out of. * Rename abiFilter to indicate it support multiple flags * Set an optimization level, it is required when enabling LTO. * Bump core commit * Bump monorepo core version * rollback mongodb realm server version * Fix checkstyle step in Jenkins file * remove the gradle logic to download core this should be moved to CMake Co-authored-by: Clemente Tort <clementetb@gmail.com> Co-authored-by: Christian Melchior <christian@ilios.dk>

# Conflicts: # realm/realm-library/src/main/cpp/object-store

…_object_accessor

…with LTO enabled on release.

Enable LTO optimizations by default

Fix gradle not picking up boolean values

fealebenpae

Looks good, great job!

realm/realm-library/src/main/cpp/CMakeLists.txt

Co-authored-by: Yavor Georgiev <fealebenpae@users.noreply.github.com>

clementetb · 2020-12-15T10:25:36Z

Sizes for generated JNI binaries:

// Unzip AAR and run 
// find . -name "*.so" -print0 | xargs -0 stat -f '%z %N'

~~ Consuming generated core artifacts ~~
Object server
5804092 ./armeabi-v7a/librealm-jni.so
8822288 ./x86/librealm-jni.so
8244096 ./arm64-v8a/librealm-jni.so
9096584 ./x86_64/librealm-jni.so
-----------

~~ Building core from source, LTO disabled ~~
Base
4342636 ./armeabi-v7a/librealm-jni.so
6516976 ./x86/librealm-jni.so
6132184 ./arm64-v8a/librealm-jni.so
6816712 ./x86_64/librealm-jni.so
-----------
Object server
5808188 ./armeabi-v7a/librealm-jni.so
8826384 ./x86/librealm-jni.so
8244096 ./arm64-v8a/librealm-jni.so
9096584 ./x86_64/librealm-jni.so
-----------

~~ Building core from source, LTO enabled ~~
Base
4313964 ./armeabi-v7a/librealm-jni.so
6275256 ./x86/librealm-jni.so
5780064 ./arm64-v8a/librealm-jni.so
6644552 ./x86_64/librealm-jni.so
-----------
Object server
5644348 ./armeabi-v7a/librealm-jni.so
8261064 ./x86/librealm-jni.so
7601168 ./arm64-v8a/librealm-jni.so
8637688 ./x86_64/librealm-jni.so
-----------

~~ Building core from source, query parser enabled, LTO disabled ~~
Base
4342636 ./armeabi-v7a/librealm-jni.so
6516976 ./x86/librealm-jni.so
6132184 ./arm64-v8a/librealm-jni.so
6816712 ./x86_64/librealm-jni.so
-----------
Object server
5808188 ./armeabi-v7a/librealm-jni.so
8826384 ./x86/librealm-jni.so
8244096 ./arm64-v8a/librealm-jni.so
9096584 ./x86_64/librealm-jni.so
-----------

cmelchior

Looks good. I'm a bit concerned about the number of ignored tests though. It could indicate we should wait with merging this until that has been fixed?

cmelchior · 2020-12-16T09:31:23Z

realm/realm-library/src/androidTest/java/io/realm/ManagedOrderedRealmCollectionTests.java

@@ -113,6 +114,7 @@
 */

 @RunWith(Parameterized.class)
+@Ignore("Tests crash due to bug in core, see https://jira.mongodb.org/browse/RCORE-435")


Is this still a problem? At least the comments in the issue seems to indicate it has been fixed?

If it hasn't been fixed, how critical is it?

It has been fixed but hasn't been released yet. It is on the core develop branch.

The issue crashes the test process, and it would crash any app. We shouldn't release without fixing the issue.

Okay, in that case, I would put this PR on hold until we can get a fix from Core. But otherwise I would say this PR is ready to be merge.

cmelchior · 2020-12-16T09:31:59Z

realm/realm-library/src/androidTest/java/io/realm/RealmJsonTests.java

@@ -75,6 +75,7 @@
 import static org.junit.Assume.assumeThat;

 @RunWith(AndroidJUnit4.class)
+@Ignore("Tests crash due to bug in core, see https://jira.mongodb.org/browse/RCORE-435")


Is this still relevant?

cmelchior · 2020-12-16T09:32:17Z

realm/realm-library/src/androidTest/java/io/realm/RealmResultsTests.java

@@ -68,6 +68,7 @@
 import static org.junit.Assert.fail;

 @RunWith(AndroidJUnit4.class)
+@Ignore("Tests crash due to bug in core, see https://jira.mongodb.org/browse/RCORE-435")


cmelchior · 2020-12-16T09:32:22Z

realm/realm-library/src/androidTest/java/io/realm/RealmTests.java

@@ -135,6 +135,7 @@


 @RunWith(AndroidJUnit4.class)
+@Ignore("Tests crash due to bug in core, see https://jira.mongodb.org/browse/RCORE-435")


cmelchior · 2020-12-16T09:38:42Z

CHANGELOG.md

@@ -1,3 +1,25 @@
+## 10.3.0 (YYYY-MM-DD)
+
+### Deprecated


We normally leave this out if nothing here.

cmelchior · 2020-12-16T09:38:53Z

CHANGELOG.md

+### Deprecated
+* None.
+
+### Breaking Changes


We normally leave this section out if nothing here.

cmelchior · 2020-12-16T09:38:59Z

CHANGELOG.md

+* None.
+
+### Compatibility
+* None.


Copy the section from previous releases here.

rorbech

Had some minor comments.

rorbech · 2020-12-21T07:39:07Z

Jenkinsfile

@@ -91,6 +91,7 @@ try {
          //  But still build all ABI's and run all types of tests. 
          useEmulator = true
          emulatorImage = "system-images;android-29;default;x86"
+          buildFlags = "-PenableLTO=true -PbuildCore=true"


For traceability/reproducability etc. I would assume that it safest to do release from the prebuilts. Is it safe to to the release from locally build cores? Or don't we get the benefits of optimizations unless we build it?

rorbech · 2020-12-21T07:42:08Z

README.md

 * Install CMake version 3.18.2 and build Ninja.
- * Install the NDK (currently r21) from the SDK Manager in Android Studio or using the [website](https://developer.android.com/ndk/downloads). If downloaded 
-You may unzip the file wherever you choose.  For macOS, a suggested location is `~/Library`.  The download will unzip as the directory `android-ndk-r21`.
+ * Install the NDK (Side-by-side) **21.0.6113669** from the SDK Manager in Android Studio. Remember to check `☑  Show package details` in the manager to display all available versions.


Isn't this download automatically when we specify the ndkVersion through the Android Gradle Plugin.

rorbech · 2020-12-21T07:44:12Z

README.md

    ```

- * If you'd like to specify the location in which to store the archives of Realm Core, define the `REALM_CORE_DOWNLOAD_DIR` environment variable. It enables you to keep Core's archive when executing `git clean -xfd`.
+ * If you'd like to specify the location in which to store the archives of Realm Core, define the `REALM_CORE_DOWNLOAD_DIR` environment variable. It enables caching core release artifacts.


Suggested change

* If you'd like to specify the location in which to store the archives of Realm Core, define the `REALM_CORE_DOWNLOAD_DIR` environment variable. It enables caching core release artifacts.

* If you'd like to specify the location in which to store the archives of Realm Core, define the `REALM_CORE_DOWNLOAD_DIR` environment variable. It enables caching of downloaded core release artifacts.

cmelchior · 2021-01-08T13:46:43Z

CHANGELOG.md

+* None.
+
+### Fixes
+* None.


We should probably add a comment about https://jira.mongodb.org/browse/HELP-21241. Maybe something like * [RealmApp] Sometimes getting Index out of range error when integrating a remote Sync changeset into the local Realm

It is a bit vague, but not sure I can come up with something more specific that doesn't involve writing 10s of lines of text.

cmelchior

If CI is happy 🚀

CHANGELOG.md

Co-authored-by: Christian Melchior <christian@ilios.dk>

cmelchior · 2021-01-08T15:08:32Z

CHANGELOG.md

+* None.
+
+### Fixes
+* [RealmApp] Integrating a remote Sync changeset into the local Realm could result in an `Index out of range error`.


It looks like the Core release has a few other fixes we should also mention. Sorry for not catching this earlier. https://github.com/realm/realm-core/blob/master/CHANGELOG.md#1033-release-notes

I would propose changing it to this:

* [RealmApp] Integrating a remote Sync changeset into the local Realm could result in an `Index out of range error`. * Change notifications not firing when removing and adding an object with the same primary key within a transaction (Issue [#7098](https://github.com/realm/realm-java/issues/7098)). * Race condition which would lead to "uncaught exception in notifier thread: N5realm15InvalidTableRefE: transaction_ended" and a crash when the source Realm was closed or invalidated at a very specific time during the first run of a collection notifier (Core issue [#3761](https://github.com/realm/realm-core/issues/3761), since v7.0.0). * Deleting and recreating objects with embedded objects could fail (Core issue [#4240](https://github.com/realm/realm-core/pull/4240), since v10.0.0)

But feel free to do that after CI has run, no reason to retrigger it for a doc update.

# Conflicts: # CHANGELOG.md # realm/realm-library/src/main/cpp/object-store

fealebenpae and others added 6 commits November 9, 2020 13:08

Update license. Remove Realm Component license. (#7194)

d80a23e

Merge branch 'master' into support-new-datatypes

fad5fdc

# Conflicts: # realm/realm-library/src/main/cpp/object-store

Added temporarily missing dictionary and mixed type functions to java…

64fd2b1

…_object_accessor

Merge master in

87f2439

Add support for consuming realm-core binaries

40c4051

clementetb self-assigned this Dec 9, 2020

clementetb added 6 commits December 9, 2020 19:10

Force building from sources on base

d10ecc7

Rename LTO switch property

554d677

Update Jenkins file to consume binaries when testing, and build core …

2f79643

…with LTO enabled on release.

Allow to cache binary resources

cd63a17

Enable LTO optimizations by default

Use system env variable for defining the core download dir

8e0e3ca

Fix gradle not picking up boolean values

Update readme

87664eb

clementetb requested review from fealebenpae, cmelchior and rorbech December 10, 2020 15:04

fealebenpae approved these changes Dec 10, 2020

View reviewed changes

realm/realm-library/src/main/cpp/CMakeLists.txt Outdated Show resolved Hide resolved

realm/realm-library/src/main/cpp/CMakeLists.txt Outdated Show resolved Hide resolved

realm/realm-library/src/main/cpp/CMakeLists.txt Outdated Show resolved Hide resolved

clementetb and others added 12 commits December 10, 2020 17:17

Update realm/realm-library/src/main/cpp/CMakeLists.txt

3e9147f

Co-authored-by: Yavor Georgiev <fealebenpae@users.noreply.github.com>

Update realm/realm-library/src/main/cpp/CMakeLists.txt

34005b5

Co-authored-by: Yavor Georgiev <fealebenpae@users.noreply.github.com>

Update realm/realm-library/src/main/cpp/CMakeLists.txt

6f35059

Co-authored-by: Yavor Georgiev <fealebenpae@users.noreply.github.com>

Add SHA validation for core artifacts

10ff4c0

Bump realm core commit

4899174

Update changelog

a11cf82

Fix rename of REALM_SYNC

ef91bb6

Remove fingerprint check

39fa223

Bump realm-core commit to master 10.3.2

271592c

Ignore failing tests

1a7f797

Ignore further failing tests

1dd23de

Profile JNI library sizes.

e9c7ea2

cmelchior approved these changes Dec 16, 2020

View reviewed changes

Update changelog

9e034aa

rorbech reviewed Dec 21, 2020

View reviewed changes

clementetb added 3 commits January 8, 2021 12:18

Bump master and enable tests affected by RCORE-435

905ca19

Fix object store not copying data

894c820

Remove ignore comments

0f41646

cmelchior reviewed Jan 8, 2021

View reviewed changes

clementetb linked an issue Jan 8, 2021 that may be closed by this pull request

RealmChangeListener not triggered, after object was deleted and replaced with same primary key. #7098

Closed

PR changes and bump core to v10.3.3

427d88f

cmelchior approved these changes Jan 8, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

clementetb and others added 3 commits January 8, 2021 15:40

Update CHANGELOG.md

c3b99c7

Co-authored-by: Christian Melchior <christian@ilios.dk>

Update CHANGELOG.md

0800bd0

Co-authored-by: Christian Melchior <christian@ilios.dk>

Point to realm core in dependencies comments

cb4a3c2

cmelchior approved these changes Jan 8, 2021

View reviewed changes

Update changelog with some more fixes

6d6e05e

clementetb changed the base branch from core-monorepo to master January 8, 2021 16:19

Merge branch 'master' into ct/consume-monorepo-binaries

31c45fb

# Conflicts: # CHANGELOG.md # realm/realm-library/src/main/cpp/object-store

clementetb merged commit b23e73e into master Jan 8, 2021

clementetb deleted the ct/consume-monorepo-binaries branch January 8, 2021 16:30

github-actions bot locked as resolved and limited conversation to collaborators Mar 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consume monorepo core binaries #7241

Consume monorepo core binaries #7241

clementetb commented Dec 9, 2020 •

edited

Loading

fealebenpae left a comment

clementetb commented Dec 15, 2020

cmelchior left a comment

cmelchior Dec 16, 2020

clementetb Dec 16, 2020 •

edited

Loading

cmelchior Dec 17, 2020

cmelchior Dec 16, 2020

cmelchior Dec 16, 2020

cmelchior Dec 16, 2020

cmelchior Dec 16, 2020

cmelchior Dec 16, 2020

cmelchior Dec 16, 2020

rorbech left a comment

rorbech Dec 21, 2020

rorbech Dec 21, 2020

rorbech Dec 21, 2020

cmelchior Jan 8, 2021

cmelchior left a comment

cmelchior Jan 8, 2021

		@@ -135,6 +135,7 @@


		@RunWith(AndroidJUnit4.class)
		@Ignore("Tests crash due to bug in core, see https://jira.mongodb.org/browse/RCORE-435")

	* If you'd like to specify the location in which to store the archives of Realm Core, define the `REALM_CORE_DOWNLOAD_DIR` environment variable. It enables caching core release artifacts.
	* If you'd like to specify the location in which to store the archives of Realm Core, define the `REALM_CORE_DOWNLOAD_DIR` environment variable. It enables caching of downloaded core release artifacts.

Consume monorepo core binaries #7241

Consume monorepo core binaries #7241

Conversation

clementetb commented Dec 9, 2020 • edited Loading

fealebenpae left a comment

Choose a reason for hiding this comment

clementetb commented Dec 15, 2020

cmelchior left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clementetb Dec 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rorbech left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmelchior left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clementetb commented Dec 9, 2020 •

edited

Loading

clementetb Dec 16, 2020 •

edited

Loading