Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup bigtable open resources on teardown #30453

Merged
merged 1 commit into from
Mar 5, 2024

Conversation

RustedBones
Copy link
Contributor

We've noticed that a failure to process a bigquery Mutation (for instance due to a coder exception), results with the following error in the logs:

Batcher was not closed properly!!! Make sure to call close().
java.lang.RuntimeException: Batcher allocation site
	at com.google.api.gax.batching.BatcherImpl$BatcherReference.<init>(BatcherImpl.java:552)
	at com.google.api.gax.batching.BatcherImpl.<init>(BatcherImpl.java:210)
	at com.google.cloud.bigtable.data.v2.stub.EnhancedBigtableStub.newMutateRowsBatcher(EnhancedBigtableStub.java:690)
	at com.google.cloud.bigtable.data.v2.BigtableDataClient.newBulkMutationBatcher(BigtableDataClient.java:1113)
	at com.google.cloud.bigtable.data.v2.BigtableDataClient.newBulkMutationBatcher(BigtableDataClient.java:1080)
	at org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl$BigtableWriterImpl.<init>(BigtableServiceImpl.java:501)
	at org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.openForWriting(BigtableServiceImpl.java:131)
	at org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.openForWriting(BigtableServiceImpl.java:93)
	at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableWriterFn.startBundle(BigtableIO.java:1248)
	at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableWriterFn$DoFnInvoker.invokeStartBundle(Unknown Source)
	at org.apache.beam.fn.harness.FnApiDoFnRunner.startBundle(FnApiDoFnRunner.java:797)
	at org.apache.beam.fn.harness.data.PTransformFunctionRegistry.lambda$register$0(PTransformFunctionRegistry.java:116)
	at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:530)
	at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)
	at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

The bigtableWriter and serviceEntry resources are allocated in the StartBundle and only closed in the FinishBundle. If an exception is raised during the ProcessElement, FinishBunlde is not called, resulting in a memory leak when retrying the step.

This change makes sure to close those resources during the TearDown phase

Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @bvolpato for label java.
R: @ahmedabu98 for label io.
R: @mutianf for label bigtable.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Copy link
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you!

@Abacn Abacn merged commit 6f8f12a into apache:master Mar 5, 2024
17 checks passed
MelodyShen pushed a commit to MelodyShen/beam that referenced this pull request Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants