Consider using google-cloud-bigquery library instead of google-api-services-bigquery

[clairemcginty]

Google documentation recommends using the client library google-cloud-bigquery rather than the API library google-api-services-bigquery.

Pros

• google-cloud-bigquery uses typed protobuf API request/response params rather than plain strings/ints, implicitly handles transport layer configurations, and potentially improves performance by making direct RPC calls rather than JSON over HTTP. We've seen intermittent failures with the scio-bigquery IT suite due to network timeouts, which might be solved by migrating.
• the google-api-services-bigquery library is in maintenance mode and aside from critical bug fixes, won't have any new features added.

Cons

• Unfortunately, the data models are quite different, and the three classes from the API library that we publicly expose in Scio - TableSchema, TableReference, and TableRow - map to Schema, TableId, and FieldValueList in the client library. So, if we end up migrating, we'd have to decide whether to change the externally facing Scio API or handle those conversions ourselves in private methods. I have a WIP branch for this migration I'll link to as soon as it's cleaned up.
• While I was developing that branch I found an issue with the client library that breaks cross-project extraction jobs: ExtractJobConfiguration's setProjectId makes cross-project BQ extracts impossible googleapis/google-cloud-java#3924 , so in its current state, client library is not fully usable in Scio.

[clairemcginty]

Update: the client library bug affecting extract jobs has been fixed! googleapis/google-cloud-java#3924

[nevillelyh]

@clairemcginty is this still worth looking?

[nevillelyh]

Talked IRL, closing.

[regadas]

I would like us to reconsider re-opening this. I think there's still some subtle bugs in our current internal BigQuery client. Some of these bugs are related to not fallbacking to env setting properties.

[regadas]

@nevillelyh @clairemcginty what was the reason to not go forward with this?

[clairemcginty]

@regadas If I remember right, it was due to the complexity of integrating with Beam's BigQuery sources/sinks -- Beam returned types from google-api-services-bigquery and a lot of the Google library functions that could convert those to google-cloud-bigquery types were private.

This was awhile ago though, so maybe worth a second look?

[regadas]

@clairemcginty interesting! I think it's worth looking into it again since we are already using the storage impl to actually retrieve data.

Let's see if the other types are good to go as well. I'll book some time to look into this.

Thanks