-
Notifications
You must be signed in to change notification settings - Fork 310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BigQueryTemplate.writeJsonStream API throwing java.lang.OutOfMemoryError: unable to create new native thread #1599
Comments
Hi @pradeepnr Thank you filing this issue! A few questions to help us understand this better:
|
Hi @mpeddada1,
|
Thank you for the background!
|
cc: @prash-mi |
@pradeepnr What I would recommend is:
This API may not be the best choice for executing too many smaller, concurrent batches. |
@prash-mi Have you considered the option of fixing the library such that it uses a thread pool? It's a best practice that would prevent this kind of issue and we do it everywhere else in the library. |
@prash-mi, Thanks for the update. |
Right @meltsufin , we actually had plans to implement thread-pool with this API but couldn't yet priortize it. Let me update this thread back when that implemented. @pradeepnr In the meantime you may want to use |
Describe the bug
Problem : We are processing huge number of requests in the range of 100s of millions. Then pushing the records to Bigquery using the BigQueryTempate::writeJsonStream API. After running for more than 2 hours and calling the API 1000 times, it is throwing the below exception
Library
spring-cloud-gcp-starter-bigquery
Bom
Details about the run
We are receiving the messages from another system and processing it and writing it to BigQuery in batch of 5000.
After 700-800 API calls we are receiving the out of memory error.
The text was updated successfully, but these errors were encountered: