Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: add heap benchmark and reduce allocations #1156

Merged
merged 10 commits into from
Oct 15, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
216 changes: 216 additions & 0 deletions benchmark.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
[INFO] Scanning for projects...
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't care if we commit this or not, but I think we should for records.

Downloading from central: https://repo.maven.apache.org/maven2/org/cyclonedx/cyclonedx-maven-plugin/2.9.0/cyclonedx-maven-plugin-2.9.0.pom
Progress (1): 1.4/19 kBProgress (1): 2.8/19 kBProgress (1): 4.1/19 kBProgress (1): 5.5/19 kBProgress (1): 6.9/19 kBProgress (1): 8.3/19 kBProgress (1): 9.7/19 kBProgress (1): 11/19 kB Progress (1): 12/19 kBProgress (1): 14/19 kBProgress (1): 15/19 kBProgress (1): 17/19 kBProgress (1): 18/19 kBProgress (1): 19 kB Downloaded from central: https://repo.maven.apache.org/maven2/org/cyclonedx/cyclonedx-maven-plugin/2.9.0/cyclonedx-maven-plugin-2.9.0.pom (19 kB at 97 kB/s)
[INFO]
[INFO] ------------------------< dev.openfeature:sdk >-------------------------
[INFO] Building OpenFeature Java SDK 1.12.0
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] Parameter 'encoding' is unknown for plugin 'maven-checkstyle-plugin:3.5.0:check (validate)'
[INFO]
[INFO] >>> jmh:0.2.2:benchmark (default-cli) > process-test-resources @ sdk >>>
[INFO]
[INFO] --- checkstyle:3.5.0:check (validate) @ sdk ---
[INFO] Starting audit...
Audit done.
[INFO] You have 0 Checkstyle violations.
[INFO]
[INFO] --- jacoco:0.8.12:prepare-agent (prepare-agent) @ sdk ---
[INFO] surefireArgLine set to -javaagent:/home/todd/.m2/repository/org/jacoco/org.jacoco.agent/0.8.12/org.jacoco.agent-0.8.12-runtime.jar=destfile=/home/todd/git/java-sdk/target/coverage-reports/jacoco-ut.exec
[INFO]
[INFO] --- resources:3.3.1:resources (default-resources) @ sdk ---
[INFO] skip non existing resourceDirectory /home/todd/git/java-sdk/src/main/resources
[INFO]
[INFO] --- compiler:3.13.0:compile (default-compile) @ sdk ---
[INFO] Nothing to compile - all classes are up to date.
[INFO]
[INFO] --- resources:3.3.1:testResources (default-testResources) @ sdk ---
[INFO] Copying 2 resources from src/test/resources to target/test-classes
[INFO]
[INFO] <<< jmh:0.2.2:benchmark (default-cli) < process-test-resources @ sdk <<<
[INFO]
[INFO]
[INFO] --- jmh:0.2.2:benchmark (default-cli) @ sdk ---
[INFO] Nothing to compile - all classes are up to date
[INFO] Executing the JMH benchmarks
# JMH version: 1.37
# VM version: JDK 21.0.4, OpenJDK 64-Bit Server VM, 21.0.4+7
# VM invoker: /usr/lib/jvm/java-21-openjdk/bin/java
# VM options: -Xmx1024m -XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC -Xmx1024m -XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: <none>
# Measurement: 1 iterations, single-shot each
# Timeout: 10 min per iteration
# Threads: 1 thread
# Benchmark mode: Single shot invocation time
# Benchmark: dev.openfeature.sdk.benchmark.AllocationBenchmark.run

# Run progress: 0.00% complete, ETA 00:00:00
# Fork: 1 of 1
[0.001s][warning][gc,init] Consider setting -Xms equal to -Xmx to avoid resizing hiccups
[0.001s][warning][gc,init] Consider enabling -XX:+AlwaysPreTouch to avoid memory commit hiccups
Iteration 1: num #instances #bytes class name (module)
-------------------------------------------------------
1: 1146018 55008864 java.util.HashMap (java.base@21.0.4)
2: 700056 11200896 java.util.HashMap$EntrySet (java.base@21.0.4)
3: 47755 9296008 [B (java.base@21.0.4)
4: 305988 8105728 [Ljava.lang.Object; (java.base@21.0.4)
5: 490930 7854880 dev.openfeature.sdk.ImmutableStructure
6: 480930 7694880 dev.openfeature.sdk.ImmutableContext
7: 100000 4000000 dev.openfeature.sdk.HookContext
8: 100000 4000000 dev.openfeature.sdk.HookContext$HookContextBuilder
9: 131512 3156288 java.util.ArrayList (java.base@21.0.4)
10: 50000 2000000 dev.openfeature.sdk.FlagEvaluationDetails
11: 50000 2000000 dev.openfeature.sdk.ProviderEvaluation
12: 153 1731448 [Ljdk.internal.vm.FillerElement; (java.base@21.0.4)
13: 51675 1653600 java.util.ArrayList$Itr (java.base@21.0.4)
14: 50002 1600064 java.util.Collections$UnmodifiableMap (java.base@21.0.4)
15: 100001 1600016 dev.openfeature.sdk.NoOpProvider$$Lambda/0x00007b6ff402fa78
16: 50000 1600000 [Ljava.util.List; (java.base@21.0.4)
17: 100000 1600000 dev.openfeature.sdk.ImmutableMetadata
18: 100000 1600000 dev.openfeature.sdk.ImmutableMetadata$ImmutableMetadataBuilder
19: 100000 1600000 dev.openfeature.sdk.OpenFeatureClient$$Lambda/0x00007b6ff40821f8
20: 75022 1200352 java.util.Optional (java.base@21.0.4)
21: 50000 1200000 dev.openfeature.sdk.FlagEvaluationOptions
22: 35143 843432 dev.openfeature.sdk.FlagEvaluationOptions$FlagEvaluationOptionsBuilder
23: 4489 679248 [I (java.base@21.0.4)
24: 26553 637272 java.lang.String (java.base@21.0.4)
25: 13155 631440 dev.openfeature.sdk.FlagEvaluationDetails$FlagEvaluationDetailsBuilder
26: 14195 567800 dev.openfeature.sdk.ProviderEvaluation$ProviderEvaluationBuilder
27: 30850 493600 dev.openfeature.sdk.internal.AutoCloseableReentrantReadWriteLock$$Lambda/0x00007b6ff402eae8
28: 18779 450696 dev.openfeature.sdk.HookSupport$$Lambda/0x00007b6ff4081230
29: 18171 436104 dev.openfeature.sdk.HookSupport$$Lambda/0x00007b6ff4081000
30: 1461 390008 [J (java.base@21.0.4)
31: 2355 288104 java.lang.Class (java.base@21.0.4)
32: 4610 258160 jdk.internal.org.objectweb.asm.SymbolTable$Entry (java.base@21.0.4)
33: 10001 240024 java.lang.Double (java.base@21.0.4)
34: 7153 228896 java.util.HashMap$EntryIterator (java.base@21.0.4)
35: 2502 180144 java.lang.reflect.Field (java.base@21.0.4)
36: 10000 160000 dev.openfeature.sdk.Value
37: 6003 144072 java.lang.StringBuilder (java.base@21.0.4)
38: 179 139928 [Ljdk.internal.org.objectweb.asm.SymbolTable$Entry; (java.base@21.0.4)
39: 3821 122272 java.util.concurrent.ConcurrentHashMap$Node (java.base@21.0.4)
40: 48 122168 [C (java.base@21.0.4)
41: 1440 113512 [S (java.base@21.0.4)
42: 1201 105688 java.lang.reflect.Method (java.base@21.0.4)
43: 3024 79424 [Ljava.lang.Class; (java.base@21.0.4)
44: 1349 75544 jdk.internal.org.objectweb.asm.Label (java.base@21.0.4)
45: 332 74368 jdk.internal.org.objectweb.asm.MethodWriter (java.base@21.0.4)
46: 1548 74304 java.lang.invoke.MemberName (java.base@21.0.4)
47: 1789 71560 java.lang.invoke.MethodType (java.base@21.0.4)
48: 1089 69696 java.net.URL (java.base@21.0.4)
49: 2011 64352 java.util.HashMap$Node (java.base@21.0.4)
50: 121 50512 [Ljava.util.concurrent.ConcurrentHashMap$Node; (java.base@21.0.4)
51: 3131 50096 jdk.internal.util.StrongReferenceKey (java.base@21.0.4)
52: 491 49608 [Ljava.util.HashMap$Node; (java.base@21.0.4)
53: 1057 42280 java.io.ObjectStreamField (java.base@21.0.4)
54: 1225 39200 java.io.File (java.base@21.0.4)
55: 779 37392 jdk.internal.org.objectweb.asm.Frame (java.base@21.0.4)
56: 243 25272 java.util.jar.JarFile$JarFileEntry (java.base@21.0.4)
57: 793 25224 [Ljava.lang.String; (java.base@21.0.4)
58: 622 24880 java.lang.NoSuchFieldException (java.base@21.0.4)
59: 571 22840 java.util.LinkedHashMap$Entry (java.base@21.0.4)
60: 473 22704 jdk.internal.ref.CleanerImpl$PhantomCleanableRef (java.base@21.0.4)
61: 687 21984 jdk.internal.util.WeakReferenceKey (java.base@21.0.4)
62: 824 19776 jdk.internal.org.objectweb.asm.ByteVector (java.base@21.0.4)
63: 248 18848 [Ljava.lang.ref.SoftReference; (java.base@21.0.4)
64: 117 17784 jdk.internal.org.objectweb.asm.ClassWriter (java.base@21.0.4)
65: 380 16824 [Ljava.lang.invoke.LambdaForm$Name; (java.base@21.0.4)
66: 625 15000 java.lang.Long (java.base@21.0.4)
67: 463 14816 java.lang.invoke.LambdaForm$Name (java.base@21.0.4)
68: 903 14448 java.lang.Object (java.base@21.0.4)
69: 198 14256 java.lang.reflect.Constructor (java.base@21.0.4)
70: 249 13944 java.util.zip.ZipFile$ZipFileInputStream (java.base@21.0.4)
71: 334 13360 jdk.internal.org.objectweb.asm.Handler (java.base@21.0.4)
72: 202 12928 java.util.concurrent.ConcurrentHashMap (java.base@21.0.4)
73: 201 12864 jdk.internal.org.objectweb.asm.FieldWriter (java.base@21.0.4)
74: 316 12640 java.util.WeakHashMap$Entry (java.base@21.0.4)
75: 102 12240 java.io.ObjectStreamClass (java.base@21.0.4)
76: 249 11952 java.util.zip.ZipFile$ZipFileInflaterInputStream (java.base@21.0.4)
77: 359 11488 jdk.internal.org.objectweb.asm.Type (java.base@21.0.4)
78: 464 11136 jdk.internal.org.objectweb.asm.Edge (java.base@21.0.4)
79: 463 11112 java.lang.invoke.ResolvedMethodName (java.base@21.0.4)
80: 341 10912 jdk.internal.math.FDBigInteger (java.base@21.0.4)
81: 94 10728 [Ljava.lang.reflect.Field; (java.base@21.0.4)
82: 266 10640 java.lang.NoSuchMethodException (java.base@21.0.4)
83: 266 10640 java.security.CodeSource (java.base@21.0.4)
84: 264 10560 sun.security.util.KnownOIDs (java.base@21.0.4)
85: 218 10464 java.lang.invoke.DirectMethodHandle$Constructor (java.base@21.0.4)
86: 75 10200 sun.nio.fs.UnixFileAttributes (java.base@21.0.4)
87: 123 9840 jdk.internal.event.DeserializationEvent (java.base@21.0.4)
88: 245 9800 java.lang.ref.SoftReference (java.base@21.0.4)
89: 115 9200 [Ljava.util.WeakHashMap$Entry; (java.base@21.0.4)
90: 368 8832 java.lang.module.ModuleDescriptor$Exports (java.base@21.0.4)
91: 63 8384 [Ljava.lang.invoke.MethodHandle; (java.base@21.0.4)
92: 146 8176 java.io.FileCleanable (java.base@21.0.4)
93: 125 8000 java.lang.Class$ReflectionData (java.base@21.0.4)
94: 322 7728 java.util.ImmutableCollections$Set12 (java.base@21.0.4)
95: 120 7680 jdk.internal.org.objectweb.asm.SymbolTable (java.base@21.0.4)
96: 69 7176 java.lang.invoke.InnerClassLambdaMetafactory (java.base@21.0.4)
97: 144 6912 jdk.internal.org.objectweb.asm.AnnotationWriter (java.base@21.0.4)
98: 167 6680 jdk.internal.loader.URLClassPath$JarLoader$2 (java.base@21.0.4)
99: 196 6272 java.lang.invoke.MethodHandles$Lookup (java.base@21.0.4)
100: 156 6240 java.util.StringJoiner (java.base@21.0.4)
101: 153 6120 java.io.FileDescriptor (java.base@21.0.4)
102: 126 6048 java.lang.invoke.LambdaForm (java.base@21.0.4)
103: 77 6016 [Ljava.lang.reflect.Method; (java.base@21.0.4)
104: 249 5976 java.util.zip.ZipFile$InflaterCleanupAction (java.base@21.0.4)
105: 370 5920 java.lang.Byte (java.base@21.0.4)
106: 74 5920 java.util.zip.ZipFile$Source (java.base@21.0.4)
107: 82 5720 [Ljava.io.ObjectStreamField; (java.base@21.0.4)
108: 40 5640 [Ljava.lang.ClassValue$Entry; (java.base@21.0.4)
109: 234 5616 java.util.jar.Attributes$Name (java.base@21.0.4)
110: 174 5568 java.util.concurrent.locks.ReentrantLock$NonfairSync (java.base@21.0.4)
111: 98 5488 java.lang.Module (java.base@21.0.4)
112: 219 5256 java.lang.PublicMethods$MethodList (java.base@21.0.4)
113: 65 5200 java.net.URI (java.base@21.0.4)
114: 215 5104 [Ljdk.internal.org.objectweb.asm.Type; (java.base@21.0.4)
115: 158 5056 java.lang.invoke.MethodTypeForm (java.base@21.0.4)
116: 152 4864 java.nio.file.attribute.FileTime (java.base@21.0.4)
117: 301 4816 java.util.HashSet (java.base@21.0.4)
118: 75 4800 java.util.zip.Inflater (java.base@21.0.4)
truncated...
Total 4538772 138760912

0.083 s/op
+totalAllocatedBytes: 138760912.000 bytes
+totalAllocatedInstances: 4538772.000 instances
+totalHeap: 521412608.000 bytes



Secondary result "dev.openfeature.sdk.benchmark.AllocationBenchmark.run:+totalAllocatedBytes":
138760912.000 bytes

Secondary result "dev.openfeature.sdk.benchmark.AllocationBenchmark.run:+totalAllocatedInstances":
4538772.000 instances

Secondary result "dev.openfeature.sdk.benchmark.AllocationBenchmark.run:+totalHeap":
521412608.000 bytes


# Run complete. Total time: 00:00:00

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.

Benchmark Mode Cnt Score Error Units
AllocationBenchmark.run ss 0.083 s/op
AllocationBenchmark.run:+totalAllocatedBytes ss 138760912.000 bytes
AllocationBenchmark.run:+totalAllocatedInstances ss 4538772.000 instances
AllocationBenchmark.run:+totalHeap ss 521412608.000 bytes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.068 s
[INFO] Finished at: 2024-10-10T11:49:20-04:00
[INFO] ------------------------------------------------------------------------
Expand Down
28 changes: 24 additions & 4 deletions pom.xml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>dev.openfeature</groupId>
Expand All @@ -11,7 +11,7 @@
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>${maven.compiler.source}</maven.compiler.target>
<junit.jupiter.version>5.11.2</junit.jupiter.version>
<!-- exclusion expression for e2e tests -->
<!-- exclusion expression for e2e tests -->
<testExclusions>**/e2e/*.java</testExclusions>
<module-name>${project.groupId}.${project.artifactId}</module-name>
</properties>
Expand Down Expand Up @@ -146,6 +146,13 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>1.37</version>
<scope>test</scope>
</dependency>

</dependencies>

<dependencyManagement>
Expand Down Expand Up @@ -473,7 +480,7 @@
<version>3.10.1</version>
<configuration>
<failOnWarnings>true</failOnWarnings>
<doclint>all,-missing</doclint> <!-- ignore missing javadoc, these are enforced with more customizability in the checkstyle plugin -->
<doclint>all,-missing</doclint> <!-- ignore missing javadoc, these are enforced with more customizability in the checkstyle plugin -->
</configuration>
<executions>
<execution>
Expand Down Expand Up @@ -507,6 +514,19 @@
</build>
</profile>

<profile>
<id>benchmark</id>
<build>
<plugins>
<plugin>
<groupId>pw.krejci</groupId>
<artifactId>jmh-maven-plugin</artifactId>
<version>0.2.2</version>
</plugin>
</plugins>
</build>
</profile>

<profile>
<id>e2e</id>
<properties>
Expand Down
16 changes: 14 additions & 2 deletions src/main/java/dev/openfeature/sdk/AbstractStructure.java
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,15 @@
@SuppressWarnings({ "PMD.BeanMembersShouldSerialize", "checkstyle:MissingJavadocType" })
abstract class AbstractStructure implements Structure {

protected final Map<String, Value> attributes;
private Map<String, Value> attributes;

@Override
public boolean isEmpty() {
return attributes == null || attributes.size() == 0;
}

AbstractStructure() {
this.attributes = new HashMap<>();
// intentionally don't initialize the attributes - do this lazily
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opinion: non-major performance improvement vs. less code later, less readable and possibly less bug prone in fututre?
Some references, from first search results:
1
2
3
4

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted for: #1156 (comment)

}

AbstractStructure(Map<String, Value> attributes) {
Expand All @@ -32,4 +37,11 @@ public Map<String, Object> asObjectMap() {
(accumulated, entry) -> accumulated.put(entry.getKey(), convertValue(entry.getValue())),
HashMap::putAll);
}

protected Map<String, Value> getAttributes() {
if (attributes == null) {
attributes = new HashMap<>();
}
return attributes;
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perf: lazily initialize.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes it non thread safe?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent point. I've reverted this because it's not a even a substantial savings. The bench result barely changes.

}
Loading
Loading