Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lazy collection fetching in Quarkus #663

Closed
markusdlugi opened this issue Mar 12, 2021 · 30 comments
Closed

lazy collection fetching in Quarkus #663

markusdlugi opened this issue Mar 12, 2021 · 30 comments
Assignees
Labels
bug Something isn't working

Comments

@markusdlugi
Copy link

When trying to fetch an association lazily with Mutiny in Quarkus, it doesn't work if the fetch is performed in a different session than the one used to persist the data. If the same session is used (so fetching right after persisting in the same session), it works.

Expected behavior
It works also in different sessions.

Actual behavior
An exception is thrown when trying to fetch the association:

2021-03-12 09:56:23,634 ERROR [org.jbo.res.rea.com.cor.AbstractResteasyReactiveContext] (vert.x-eventloop-thread-2) Request failed: org.hibernate.LazyInitializationException: Collection cannot be initialized: com.example.Author.books
	at org.hibernate.reactive.session.impl.ReactiveSessionImpl.initializeCollection(ReactiveSessionImpl.java:330)
	at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:589)
	at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:264)
	at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:585)
	at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:149)
	at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:178)
	at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:163)
	at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:264)
	at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:162)
	at org.hibernate.collection.internal.PersistentBag.size(PersistentBag.java:371)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.takeCollectionSizeSnapshot(LazyAttributeLoadingInterceptor.java:160)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.lambda$loadAttribute$0(LazyAttributeLoadingInterceptor.java:110)
	at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementHelper.performWork(EnhancementHelper.java:130)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.loadAttribute(LazyAttributeLoadingInterceptor.java:76)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.fetchAttribute(LazyAttributeLoadingInterceptor.java:72)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.handleRead(LazyAttributeLoadingInterceptor.java:53)
	at org.hibernate.bytecode.enhance.spi.interceptor.AbstractInterceptor.readObject(AbstractInterceptor.java:153)
	at com.example.Author.$$_hibernate_read_books(Author.java)
	at com.example.Author.getBooks(Author.java:43)
	at com.example.TestResource.lambda$getBooks$2(TestResource.java:61)
	at io.smallrye.mutiny.operators.UniOnItemTransformToUni.invokeAndSubstitute(UniOnItemTransformToUni.java:31)
	at io.smallrye.mutiny.operators.UniOnItemTransformToUni$2.onItem(UniOnItemTransformToUni.java:74)
	at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
	at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
	at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
	at io.smallrye.mutiny.operators.UniSerializedSubscriber.onItem(UniSerializedSubscriber.java:85)
	at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
	at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
	at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
	at io.smallrye.mutiny.operators.UniDelegatingSubscriber.onItem(UniDelegatingSubscriber.java:24)
	at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
	at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
	at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
	at io.smallrye.mutiny.operators.UniSerializedSubscriber.onItem(UniSerializedSubscriber.java:85)
	at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
	at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
	at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage.lambda$forwardFromCompletionStage$1(UniCreateFromCompletionStage.java:30)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at com.ibm.asyncutil.iteration.AsyncTrampoline$TrampolineInternal.unroll(AsyncTrampoline.java:127)
	at com.ibm.asyncutil.iteration.AsyncTrampoline$TrampolineInternal.lambda$unroll$0(AsyncTrampoline.java:123)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at org.hibernate.reactive.pool.impl.Handlers.lambda$toCompletionStage$0(Handlers.java:26)
	at io.vertx.sqlclient.impl.SqlResultHandler.complete(SqlResultHandler.java:98)
	at io.vertx.sqlclient.impl.SqlResultHandler.handle(SqlResultHandler.java:87)
	at io.vertx.sqlclient.impl.SqlResultHandler.handle(SqlResultHandler.java:33)
	at io.vertx.sqlclient.impl.SocketConnectionBase.handleMessage(SocketConnectionBase.java:241)
	at io.vertx.sqlclient.impl.SocketConnectionBase.lambda$init$0(SocketConnectionBase.java:88)
	at io.vertx.core.net.impl.NetSocketImpl.lambda$new$2(NetSocketImpl.java:101)
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:237)
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:127)
	at io.vertx.core.net.impl.NetSocketImpl.handleMessage(NetSocketImpl.java:357)
	at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
	at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:163)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.vertx.pgclient.impl.codec.PgEncoder.lambda$write$0(PgEncoder.java:78)
	at io.vertx.pgclient.impl.codec.PgCommandCodec.handleReadyForQuery(PgCommandCodec.java:138)
	at io.vertx.pgclient.impl.codec.PgDecoder.decodeReadyForQuery(PgDecoder.java:226)
	at io.vertx.pgclient.impl.codec.PgDecoder.channelRead(PgDecoder.java:86)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

Reproducer

https://github.com/markusdlugi/hibernate-reactive-lazy-fetching

Steps to reproduce the behavior:

  1. Start database using infrastructure/docker-compose.yml.
  2. Start Quarkus application using mvn quarkus:dev
  3. Send request: POST http://localhost:8080/test/working.
    3.1. This works and returns a list of books.
  4. Send request: GET http://localhost:8080/test/failing/1 (or other authorId created via POST http://localhost:8080/test/failing).
    4.1 This fails with the above exception.
  • Quarkus Version: 1.12.2.Final
  • Hibernate Reactive Version: 1.0.0.Beta4 (also tested with 1.0.0.CR1 by overwriting the version in Maven, same behavior)
  • Hibernate Core Version: 5.4.28.Final (also tested with 5.4.29.Final by overwriting the version in Maven, same behavior)
@markusdlugi
Copy link
Author

/cc @DavideD @gavinking

@gavinking
Copy link
Member

Yeah, I think we decided this was the correct thing to do: in JPA there's no notion of reassociating an entity instance with a session. That's a bit different to the really old usage pattern of Hibernate where you could use saveOrUpdate() to directly reassociate a detached object. In JPA the model for dealing with detached objects is the merge() operation.

(And there's good reasons to not have this. Experience is that in practice users get themselves into a mess with reassociation.)

Since loading a detached collection really implies reattaching its parent, we simply don't support this.

However, IIRC, I do let you do this in a StatelessSession, since it's much more compatible with that programming model. Try using a stateless session and let me know if it works for you.

@markusdlugi
Copy link
Author

Hi @gavinking, thanks for the quick reply.

I'm not quite sure whether I'm getting what you're saying. So what I implemented was exactly what is described in the documentation:

session.find(Author.class, authorId)
	.chain(author -> Mutiny.fetch(author.getBooks()));

This is working if the author and his books have been persisted with the same session (i.e., in the same HTTP request), but it's not working if I use another session (i.e., another request).

So you are implying that this is intended behavior, at least for a "normal" (stateful?) session? This seems to me like a really standard use case, because I will almost always try to get some data from the database some time after persisting it, so in a different session. This would mean that using a normal session, lazy associations are not supported. In that case, I would argue that it should be explicitly mentioned in the documentation that a StatelessSession needs to be used if lazy associations are to be fetched.

@gavinking
Copy link
Member

I'm saying that in the new session you should use getReference() to obtain a session-bound reference to the entity that owns the collection. Then you can fetch the collection however you like.

This is the standard JPA model, it's even a bit easier to use in HR because I overloaded getReference().

@DavideD
Copy link
Member

DavideD commented Mar 12, 2021

@gavinking do you mean something like this?

Author reference = session.getReference( Author.class, authorId );
return Mutiny.fetch( reference.getBooks() );

@gavinking
Copy link
Member

Sure, but I made it even easier. You can directly pass the detached author to getReference().

@markusdlugi
Copy link
Author

Okay, I tried those suggestions, but unfortunately they didn't work either. Maybe you can be so kind to point out if I'm doing something incorrectly here:

https://github.com/markusdlugi/hibernate-reactive-lazy-fetching/blob/f3458eee969d7c3597560aaf8a646b7aad2ee9b0/src/main/java/com/example/TestResource.java#L55-L79

The approach using getReference() fails with the following exception:

2021-03-12 14:17:51,381 ERROR [org.jbo.res.rea.ser.cor.ExceptionMapping] (vert.x-eventloop-thread-10) Request failed : java.lang.NullPointerException
	at org.hibernate.engine.internal.AbstractEntityEntry.overwriteLoadedStateCollectionValue(AbstractEntityEntry.java:336)
	at org.hibernate.persister.entity.AbstractEntityPersister.initializeLazyProperty(AbstractEntityPersister.java:1144)
	at org.hibernate.persister.entity.AbstractEntityPersister.initializeEnhancedEntityUsedAsProxy(AbstractEntityPersister.java:4497)
	at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementAsProxyLazinessInterceptor.forceInitialize(EnhancementAsProxyLazinessInterceptor.java:221)
	at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementAsProxyLazinessInterceptor.lambda$handleRead$0(EnhancementAsProxyLazinessInterceptor.java:133)
	at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementHelper.performWork(EnhancementHelper.java:130)
	at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementAsProxyLazinessInterceptor.handleRead(EnhancementAsProxyLazinessInterceptor.java:98)
	at org.hibernate.bytecode.enhance.spi.interceptor.AbstractInterceptor.readObject(AbstractInterceptor.java:153)
	at com.example.Author.$$_hibernate_read_books(Author.java)
	at com.example.Author.getBooks(Author.java:43)
	at com.example.TestResource.getBooksUsingReference(TestResource.java:69)
	...

The other suggested solution using StatelessSession fails with:

2021-03-12 14:21:11,840 ERROR [org.jbo.res.rea.ser.cor.ExceptionMapping] (vert.x-eventloop-thread-5) Request failed : org.hibernate.LazyInitializationException: Unable to perform requested lazy initialization [com.example.Author.books] - no session and settings disallow loading outside the Session
	at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementHelper.throwLazyInitializationException(EnhancementHelper.java:199)
	at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementHelper.performWork(EnhancementHelper.java:89)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.loadAttribute(LazyAttributeLoadingInterceptor.java:76)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.fetchAttribute(LazyAttributeLoadingInterceptor.java:72)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.handleRead(LazyAttributeLoadingInterceptor.java:53)
	at org.hibernate.bytecode.enhance.spi.interceptor.AbstractInterceptor.readObject(AbstractInterceptor.java:153)
	at com.example.Author.$$_hibernate_read_books(Author.java)
	at com.example.Author.getBooks(Author.java:43)
	at com.example.TestResource.lambda$getBooksUsingStatelessSession$3(TestResource.java:78)
	...

@DavideD
Copy link
Member

DavideD commented Mar 12, 2021

I will have a look, it seems related to bytecode enhancements

@markusdlugi
Copy link
Author

@DavideD, any update?

I tried to reproduce the issue without Quarkus by just using the session-example in this repo, but I was unable to do so. With just Hibernate Reactive and plain Mutiny, it works as expected. It also works using find(), no getReference() or StatelessSession necessary.

So this issue only seems to happen in Quarkus.

@markusdlugi markusdlugi changed the title Lazy association fetching doesn't work in different sessions Lazy association fetching doesn't work in Quarkus Mar 15, 2021
@DavideD
Copy link
Member

DavideD commented Mar 15, 2021

I'm still looking into it.

The problem happens because of bytecode enhancements that are enabled by default in Quarkus.
I haven't still figure out which set of properties recreate the issue when using only Hibernate Reactive. But I will continue to work on it today

@DavideD
Copy link
Member

DavideD commented Mar 16, 2021

I can now recreate the error in reactive. It requires two things:

  1. set hibernate.bytecode.allow_enhancement_as_proxy to true
  2. set SessionFactoryOptions#enableCollectionInDefaultFetchGroup to false. It should be set to true:

@gavinking Am I correct in thinking that we should make it sure that in Quarkus enableCollectionInDefaultFetchGroup is set to false? I admit that at the moment I don't understand the whole issue fully (I've tried to follow the conversation on issue #374)

@gavinking
Copy link
Member

Um. I think it only makes sense to have the collection in the default fetch group. I don't think it ever makes sense to exclude it.

@DavideD
Copy link
Member

DavideD commented Mar 16, 2021

n Quarkus enableCollectionInDefaultFetchGroup is set to false

Sorry, yeah, I meant to write is set to true . Time for a break :-)

@gavinking
Copy link
Member

Ah OK good :-) Take a break :-)

@markusdlugi
Copy link
Author

@DavideD That's some great findings, thank you :)

Just wanted to point out, that according to the discussion in #374 and the related hibernate/hibernate-orm#3558, it seems like enableCollectionInDefaultFetchGroup was deliberately left to be false in Quarkus? At least that's what I'm gathering from this comment:

hibernate/hibernate-orm#3558 (comment)

Anyways, it does look like Quarkus currently doesn't set that property to true, because it is unconfigured in FastBootReactiveEntityManagerFactoryBuilder:

https://github.com/quarkusio/quarkus/blob/55cf15173ff7b2985a8422de050d1c9c708be57c/extensions/hibernate-reactive/runtime/src/main/java/io/quarkus/hibernate/reactive/runtime/boot/FastBootReactiveEntityManagerFactoryBuilder.java#L26-L32

And if I checked it correctly, the default in Hibernate ORM still seems to be false, which would explain why we see this behavior in Quarkus only.

@gavinking
Copy link
Member

@Sanne needs to look at this one.

@gavinking gavinking added the bug Something isn't working label Mar 17, 2021
@gavinking gavinking changed the title Lazy association fetching doesn't work in Quarkus hibernate.bytecode.allow_enhancement_as_proxy in Quarkus Mar 17, 2021
@DavideD
Copy link
Member

DavideD commented Mar 17, 2021

If we only have to set it to true, it seems pretty easy to solve. I will create a fix and then check with @Sanne

@gavinking
Copy link
Member

Well what I'm wondering is if it's set to false because that is what ORM needs, and if we need to be careful to not step on regular Hibernate?

@Sanne
Copy link
Member

Sanne commented Mar 17, 2021

great find, thanks. Yes it's a little tricky because the two extensions ("regular" and "reactive") are highly coupled ATM.

Need to separate them properly, that was developed in too much rush.

@gavinking
Copy link
Member

(OTOH, I'm not sure why ORM would need it; as I've argued before, I think it should be on by default.)

@Sanne
Copy link
Member

Sanne commented Mar 17, 2021

hum ok. Let's do this for both ORMs (in Quarkus).

@markusdlugi
Copy link
Author

@gavinking, I think that title is wrong, isn't it? I guess you meant to put enableCollectionInDefaultFetchGroup there instead. Bytecode enhancements are necessary for Quarkus anyway because of native, I assume :)

@gavinking
Copy link
Member

Ah yeah, sure, I guess you're right.

you're saying that you see this bug with default settings in Quarkus?

@markusdlugi
Copy link
Author

markusdlugi commented Mar 17, 2021

Yes, no special configuration is necessary in Quarkus to get this bug. @DavideD just had to do some special configuration to reproduce in plain HR (without Quarkus).

@markusdlugi markusdlugi changed the title hibernate.bytecode.allow_enhancement_as_proxy in Quarkus enableCollectionInDefaultFetchGroup in Quarkus Mar 17, 2021
@gavinking
Copy link
Member

Sure.

The thing is it's not actually a bug in HR. I deliberately added that "collection in default fetch group" thing to core specifically because we need it on all the time in HR.

So the bug is in the Quarkus extension which turns it off.

(Or, arguably, in core for doing the wrong thing by default.)

@gavinking gavinking changed the title enableCollectionInDefaultFetchGroup in Quarkus lazy collection fetching in Quarkus Mar 17, 2021
DavideD added a commit to DavideD/quarkus that referenced this issue Mar 17, 2021
The default for ORM is false, but when we create the SessionFactory
for Hibernate Reactive this value has to be true.

For more details, see:
hibernate/hibernate-reactive#663
@DavideD
Copy link
Member

DavideD commented Mar 17, 2021

I've sent a fix for Quarkus: quarkusio/quarkus#15818
getReference doesn't seem to work on quarkus at the moment. I will create a separate issue for it.

@markusdlugi
Copy link
Author

Sounds great, thank you! 👍

@markusdlugi
Copy link
Author

Just tested it and can confirm that lazy collection fetching now works in Quarkus. Thanks once again 😄

@DavideD
Copy link
Member

DavideD commented Mar 19, 2021

Thank you @markusdlugi

@DavideD DavideD closed this as completed Mar 19, 2021
@gavinking
Copy link
Member

Thanks for reporting it @markusdlugi

gsmet pushed a commit to gsmet/quarkus that referenced this issue Mar 23, 2021
The default for ORM is false, but when we create the SessionFactory
for Hibernate Reactive this value has to be true.

For more details, see:
hibernate/hibernate-reactive#663

(cherry picked from commit 76072c5)
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    The default for ORM is false, but when we create the SessionFactory
    for Hibernate Reactive this value has to be true.

    For more details, see:
    hibernate/hibernate-reactive#663
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants