15 Pull requests merged by 10 people

• [Prism] Support BundleFinalization DoFn parameter

  #32425 merged Sep 21, 2024

• [#32498][prism] Add split / progress back off + catch-up.

  #32526 merged Sep 21, 2024

• [Python] Add a couple quality-of-life improvemenets to testing.util.assert_that

  #30771 merged Sep 21, 2024

• Rollback Bigtable throttling counter

  #32442 merged Sep 20, 2024

• Remove debug leftover

  #32524 merged Sep 20, 2024

• Include jamm as runtime dependency in expansion-service

  #32521 merged Sep 20, 2024

• [Python] Fix bug where WorkerHandlerManager.close_all re-initialized state with incorrect dict type

  #32517 merged Sep 20, 2024

• Change PeriodicSequence to report backlog accurately

  #32505 merged Sep 20, 2024

• Add runners/prism/java to java PreCommit

  #32507 merged Sep 19, 2024

• Support Managed Iceberg streaming writes

  #32451 merged Sep 19, 2024

• Emit warning when Mqtt waiting for connection for extended period of time

  #32500 merged Sep 19, 2024

• Publish Java Expansion Service SNAPSHOT containers

  #32491 merged Sep 19, 2024

• Add missing test to test added commit stage is added if enabled.

  #32438 merged Sep 19, 2024

• Disable a permared suite.

  #32493 merged Sep 18, 2024

• Update python SDK container

  #32501 merged Sep 18, 2024

16 Pull requests opened by 11 people

• [Dataflow Streaming] Use isolated windmill streams based on job settings

  #32503 opened Sep 19, 2024

• Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.17.12 to 1.17.21 in /sdks

  #32504 opened Sep 19, 2024

• Fix tableExists() method

  #32510 opened Sep 19, 2024

• [Dataflow Streaming] Use separate heartbeat streams based on job settings

  #32511 opened Sep 20, 2024

• Support Map and Arrays of Maps in BQ for StorageWrites API for Beam Rows

  #32512 opened Sep 20, 2024

• [WIP] fix backlog bytes periodicsequence in python and go sdks to not include future outputs

  #32513 opened Sep 20, 2024

• BigQuery trim schema with selected fields

  #32514 opened Sep 20, 2024

• BigQuey fix invalid null checks in io translation

  #32515 opened Sep 20, 2024

• [Python] Allow ints to be considered as floats/complexes and floats as complexes

  #32516 opened Sep 20, 2024

• [GCSIO] Fix internal unit test failure

  #32518 opened Sep 20, 2024

• [yaml] Add Beam YAML Examples

  #32519 opened Sep 20, 2024

• Invoke teardown when DoFn throws in portable runners

  #32522 opened Sep 20, 2024

• [WIP] Enable CDC configuration for cross language StorageWrite API usage

  #32523 opened Sep 20, 2024

• Add schrodinger logo

  #32525 opened Sep 20, 2024

• [WIP] Enable BigQuery CDC configuration for Python BigQuery sink

  #32529 opened Sep 21, 2024

• Move assert into pipeline

  #32530 opened Sep 21, 2024

6 Issues closed by 5 people

• [Bug]: How to run YAML pipeline from Python SDK rather than from command line

  #32416 closed Sep 21, 2024

• The PostCommit TransformService Direct job is flaky

  #30960 closed Sep 21, 2024

• The Python tests job is flaky

  #31006 closed Sep 21, 2024

• [Prism] Support Bundle Finalization

  #31912 closed Sep 21, 2024

• [Bug]: minimal wordcount golang example is freezeing on gcs reading

  #32498 closed Sep 21, 2024

• MqttIOTest.testRead is flaky (sickbayed)

  #19092 closed Sep 19, 2024

8 Issues opened by 7 people

• [Feature Request][Java]:filesToStage should support the gcs file path

  #32531 opened Sep 21, 2024

• [Feature Request]: vLLM model handler

  #32528 opened Sep 21, 2024

• [Feature Request]: Enable CDC configuration for cross language StorageWrite API usage

  #32527 opened Sep 20, 2024

• [Bug]: ParDoLifeCycleTest in Stateful DoFn fails on Samza PVR runner

  #32520 opened Sep 20, 2024

• [Bug]: Unable to Restart Google Spanner Change Streams Consumer due to tableExists(table_name) bug

  #32509 opened Sep 19, 2024

• [Bug]: BigqueryIO is very slow if using storage api and dynamic destination to write data to over thousand different tables with high data skew

  #32508 opened Sep 19, 2024

• [Bug]: PeriodicSequence reports large amount of remaining work / size when used with infinite stop time

  #32506 opened Sep 19, 2024

• [Bug]: Pipeline may stuck after error thrown in transform due to racing condition in CancellableQueue

  #32502 opened Sep 18, 2024

21 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Improve BQ <-> Avro conversions

  #32482 commented on Sep 20, 2024 • 20 new comments

• ReadFromKafkaDoFn: handle failed seek

  #32456 commented on Sep 20, 2024 • 8 new comments

• Rag opensearch usecase with Beam's MLTransform

  #32018 commented on Sep 21, 2024 • 6 new comments

• Add RowFilter utility

  #32366 commented on Sep 20, 2024 • 3 new comments

• Kafka metrics

  #32402 commented on Sep 19, 2024 • 2 new comments

• [yaml] Add enrichment transform to Beam YAML

  #32286 commented on Sep 20, 2024 • 2 new comments

• Set a UUID when building a Schema object.

  #32399 commented on Sep 20, 2024 • 1 new comment

• Upgrade default kafka from 2.4.1 to 3.1.2

  #32486 commented on Sep 20, 2024 • 0 new comments

• Add various utility meta-transforms to Beam.

  #32445 commented on Sep 20, 2024 • 0 new comments

• [Prism] Remove unused code

  #32414 commented on Sep 20, 2024 • 0 new comments

• [WIP] Do not merge: Vllm model handler

  #32410 commented on Sep 21, 2024 • 0 new comments

• BigQueryIO uniformize direct and export reads

  #32360 commented on Sep 20, 2024 • 0 new comments

• [yaml] Add use cases for Enrichment transform in YAML

  #32289 commented on Sep 19, 2024 • 0 new comments

• make FieldValueTypeInformation creators take a TypeDescriptor parameter

  #32081 commented on Sep 21, 2024 • 0 new comments

• Add correct shutdown mechanics for direct path

  #31902 commented on Sep 20, 2024 • 0 new comments

• Poll for backlog in background thread instead of inline

  #31697 commented on Sep 19, 2024 • 0 new comments

• `beam.CombineValues` on DataFlow runner causes ambiguous failure with python SDK

  #21432 commented on Sep 21, 2024 • 0 new comments

• [Task]: Use "Distroless" Container Images to reduce potential vulnerabilities for Beam Python SDK containers

  #28991 commented on Sep 21, 2024 • 0 new comments

• [Tracking Umbrella] Prism Runner areas for contribution.

  #29650 commented on Sep 21, 2024 • 0 new comments

• Dataflow runner does not respect ParDo's lifecycle on case of exceptions

  #18592 commented on Sep 20, 2024 • 0 new comments

• Clickhouse test failures

  #20912 commented on Sep 19, 2024 • 0 new comments