42 Pull requests merged by 21 people

• [Prism] Support BundleFinalization DoFn parameter

  #32425 merged Sep 21, 2024

• [#32498][prism] Add split / progress back off + catch-up.

  #32526 merged Sep 21, 2024

• [Python] Add a couple quality-of-life improvemenets to testing.util.assert_that

  #30771 merged Sep 21, 2024

• Rollback Bigtable throttling counter

  #32442 merged Sep 20, 2024

• Remove debug leftover

  #32524 merged Sep 20, 2024

• Include jamm as runtime dependency in expansion-service

  #32521 merged Sep 20, 2024

• [Python] Fix bug where WorkerHandlerManager.close_all re-initialized state with incorrect dict type

  #32517 merged Sep 20, 2024

• Change PeriodicSequence to report backlog accurately

  #32505 merged Sep 20, 2024

• Add runners/prism/java to java PreCommit

  #32507 merged Sep 19, 2024

• Support Managed Iceberg streaming writes

  #32451 merged Sep 19, 2024

• Emit warning when Mqtt waiting for connection for extended period of time

  #32500 merged Sep 19, 2024

• Publish Java Expansion Service SNAPSHOT containers

  #32491 merged Sep 19, 2024

• Add missing test to test added commit stage is added if enabled.

  #32438 merged Sep 19, 2024

• Disable a permared suite.

  #32493 merged Sep 18, 2024

• Update python SDK container

  #32501 merged Sep 18, 2024

• Add throttling counter in gcsio and refactor retrying

  #32428 merged Sep 18, 2024

• Better error message for incorrect pipeline options flags.

  #32478 merged Sep 18, 2024

• Restore release-essential references to 3.8

  #32499 merged Sep 18, 2024

• Fix JmsIO NPE

  #32489 merged Sep 18, 2024

• Warn when using Beam with Python 3.8

  #32484 merged Sep 18, 2024

• [Dataflow Streaming] Add support to read user worker settings from backend

  #32408 merged Sep 18, 2024

• Bump github.com/docker/docker from 27.1.2+incompatible to 27.2.1+incompatible in /sdks

  #32495 merged Sep 18, 2024

• Update python SDK container

  #32487 merged Sep 17, 2024

• Fix non-utf8 encoding problem in ReadFromCsv and WriteToCsv.

  #32463 merged Sep 17, 2024

• Update run_inference_huggingface.ipynb

  #32488 merged Sep 17, 2024

• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.60.0 to 1.61.2 in /sdks

  #32397 merged Sep 17, 2024

• @JsonIgnore causes emulator endpoint to be missing in DoFn's

  #32450 merged Sep 17, 2024

• Bump google.golang.org/api from 0.192.0 to 0.197.0 in /sdks

  #32426 merged Sep 17, 2024

• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.28 to 1.27.34 in /sdks

  #32480 merged Sep 17, 2024

• Update Dataflow internal client test

  #32471 merged Sep 17, 2024

• Restore mypy ignore[call-arg] in yaml_io.py

  #32475 merged Sep 17, 2024

• Fix Kafka with Redistribute

  #32344 merged Sep 17, 2024

• Bump golang.org/x/sys from 0.24.0 to 0.25.0 in /sdks

  #32479 merged Sep 17, 2024

• Add information on python metrics to the programming guide

  #32464 merged Sep 16, 2024

• [yaml] Add examples for Spanner IO in YAML

  #32288 merged Sep 16, 2024

• Fix macos tox.ini definition

  #32472 merged Sep 16, 2024

• Add the script to extract docs from the dev list

  #32460 merged Sep 16, 2024

• Remove Python 3.8 GitHub Actions Runs

  #32429 merged Sep 16, 2024

• fix: ensure that all outstanding future have completed in FinishBundle

  #32454 merged Sep 16, 2024

• Bump golang.org/x/text from 0.17.0 to 0.18.0 in /sdks

  #32466 merged Sep 16, 2024

• Don't override existing consumer config values KafkaIO xlang

  #32443 merged Sep 15, 2024

• fix docker compose command

  #32459 merged Sep 14, 2024

27 Pull requests opened by 17 people

• fix: add topic existing validation

  #32465 opened Sep 15, 2024

• [DRAFT] fix new golang repo versioning and add python 3.12 venv to install th…

  #32469 opened Sep 16, 2024

• Add support for dynamic write in `MqttIO`

  #32470 opened Sep 16, 2024

• [feat 32473] Added soft deadline logic to Spanner Change Stream IO connector.

  #32474 opened Sep 16, 2024

• do not review

  #32476 opened Sep 16, 2024

• Bump dompurify from 2.3.3 to 2.5.6 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel

  #32477 opened Sep 16, 2024

• Improve BQ <-> Avro conversions

  #32482 opened Sep 17, 2024

• Upgrade default kafka from 2.4.1 to 3.1.2

  #32486 opened Sep 17, 2024

• Bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.32 to 1.17.33 in /sdks

  #32494 opened Sep 18, 2024

• Bump github.com/nats-io/nats-server/v2 from 2.10.18 to 2.10.20 in /sdks

  #32496 opened Sep 18, 2024

• Bump go.mongodb.org/mongo-driver from 1.16.1 to 1.17.0 in /sdks

  #32497 opened Sep 18, 2024

• [Dataflow Streaming] Use isolated windmill streams based on job settings

  #32503 opened Sep 19, 2024

• Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.17.12 to 1.17.21 in /sdks

  #32504 opened Sep 19, 2024

• Fix tableExists() method

  #32510 opened Sep 19, 2024

• [Dataflow Streaming] Use separate heartbeat streams based on job settings

  #32511 opened Sep 20, 2024

• Support Map and Arrays of Maps in BQ for StorageWrites API for Beam Rows

  #32512 opened Sep 20, 2024

• [WIP] fix backlog bytes periodicsequence in python and go sdks to not include future outputs

  #32513 opened Sep 20, 2024

• BigQuery trim schema with selected fields

  #32514 opened Sep 20, 2024

• BigQuey fix invalid null checks in io translation

  #32515 opened Sep 20, 2024

• [Python] Allow ints to be considered as floats/complexes and floats as complexes

  #32516 opened Sep 20, 2024

• [GCSIO] Fix internal unit test failure

  #32518 opened Sep 20, 2024

• [yaml] Add Beam YAML Examples

  #32519 opened Sep 20, 2024

• Invoke teardown when DoFn throws in portable runners

  #32522 opened Sep 20, 2024

• [WIP] Enable CDC configuration for cross language StorageWrite API usage

  #32523 opened Sep 20, 2024

• Add schrodinger logo

  #32525 opened Sep 20, 2024

• [WIP] Enable BigQuery CDC configuration for Python BigQuery sink

  #32529 opened Sep 21, 2024

• Move assert into pipeline

  #32530 opened Sep 21, 2024

11 Issues closed by 7 people

• The PostCommit TransformService Direct job is flaky

  #30960 closed Sep 21, 2024

• The Python tests job is flaky

  #31006 closed Sep 21, 2024

• [Prism] Support Bundle Finalization

  #31912 closed Sep 21, 2024

• [Bug]: minimal wordcount golang example is freezeing on gcs reading

  #32498 closed Sep 21, 2024

• MqttIOTest.testRead is flaky (sickbayed)

  #19092 closed Sep 19, 2024

• The PostCommit XVR Direct job is flaky

  #30517 closed Sep 18, 2024

• [Bug]: NPE on JmsIO

  #32483 closed Sep 18, 2024

• [Feature Request]: Improve performance of generating successful writes in Java's BigqueryIO connector

  #31918 closed Sep 18, 2024

• [Bug]: Error when using ReadFromCsv to read non-utf8 encoding file

  #32462 closed Sep 17, 2024

• Performance Regression or Improvement: test_cloudml_benchmark_criteo_10GB-runtime_sec:runtime_sec

  #32457 closed Sep 17, 2024

• [Bug]: windmillServiceCommitThreads option can lead to ConcurrentModificationException and stuck commits

  #31303 closed Sep 16, 2024

14 Issues opened by 13 people

• [Feature Request]: vLLM model handler

  #32528 opened Sep 21, 2024

• [Feature Request]: Enable CDC configuration for cross language StorageWrite API usage

  #32527 opened Sep 20, 2024

• [Bug]: ParDoLifeCycleTest in Stateful DoFn fails on Samza PVR runner

  #32520 opened Sep 20, 2024

• [Bug]: Unable to Restart Google Spanner Change Streams Consumer due to tableExists(table_name) bug

  #32509 opened Sep 19, 2024

• [Bug]: BigqueryIO is very slow if using storage api and dynamic destination to write data to over thousand different tables with high data skew

  #32508 opened Sep 19, 2024

• [Bug]: PeriodicSequence reports large amount of remaining work / size when used with infinite stop time

  #32506 opened Sep 19, 2024

• [Bug]: Pipeline may stuck after error thrown in transform due to racing condition in CancellableQueue

  #32502 opened Sep 18, 2024

• [Bug]: Re-enable GoUsingJava xlang suites

  #32492 opened Sep 18, 2024

• [Feature Request]: Add option to set maximum Storage API streams when using autosharding

  #32490 opened Sep 17, 2024

• [CsvIO]: Support non-UTF8 text encoding for parse/reads and writes

  #32485 opened Sep 17, 2024

• [Bug]: tour of beam content cannot be scrolled on ipad

  #32481 opened Sep 17, 2024

• [Feature Request]: Add soft-deadline before 1 minute hard-deadline to Spanner Change Stream IO connector

  #32473 opened Sep 16, 2024

• [Feature Request]: Topic array support for MqttIO.Read

  #32468 opened Sep 16, 2024

• [Bug]: PubsubIO on Flink Runner not acknowledging old messages

  #32461 opened Sep 15, 2024

32 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• ReadFromKafkaDoFn: handle failed seek

  #32456 commented on Sep 20, 2024 • 22 new comments

• Add RowStringInterpolator utility

  #32367 commented on Sep 18, 2024 • 15 new comments

• Add correct shutdown mechanics for direct path

  #31902 commented on Sep 20, 2024 • 11 new comments

• Support writing to Pubsub with ordering key; Add PubsubMessage SchemaCoder

  #31608 commented on Sep 16, 2024 • 7 new comments

• Add RowFilter utility

  #32366 commented on Sep 20, 2024 • 6 new comments

• Rag opensearch usecase with Beam's MLTransform

  #32018 commented on Sep 21, 2024 • 6 new comments

• SolaceIO write connector

  #32060 commented on Sep 16, 2024 • 5 new comments

• Kafka metrics

  #32402 commented on Sep 19, 2024 • 4 new comments

• Set a UUID when building a Schema object.

  #32399 commented on Sep 20, 2024 • 3 new comments

• BigQueryIO uniformize direct and export reads

  #32360 commented on Sep 20, 2024 • 3 new comments

• [yaml] Add enrichment transform to Beam YAML

  #32286 commented on Sep 20, 2024 • 2 new comments

• [Prism] Remove unused code

  #32414 commented on Sep 20, 2024 • 0 new comments

• [WIP] Do not merge: Vllm model handler

  #32410 commented on Sep 21, 2024 • 0 new comments

• [DO NOT REVIEW] Run GHA tests off master branch.

  #32424 commented on Sep 18, 2024 • 0 new comments

• [flink-runner] Improve Datastream for batch performances

  #32440 commented on Sep 16, 2024 • 0 new comments

• Fix writing raw messages to pubsub

  #32342 commented on Sep 16, 2024 • 0 new comments

• Add various utility meta-transforms to Beam.

  #32445 commented on Sep 20, 2024 • 0 new comments

• [yaml] Add use cases for Enrichment transform in YAML

  #32289 commented on Sep 19, 2024 • 0 new comments

• Bump github.com/fsouza/fake-gcs-server from 1.49.2 to 1.49.3 in /sdks

  #32194 commented on Sep 16, 2024 • 0 new comments

• make FieldValueTypeInformation creators take a TypeDescriptor parameter

  #32081 commented on Sep 21, 2024 • 0 new comments

• Fix MongoDB connector to handle out-of-range dates

  #31890 commented on Sep 18, 2024 • 0 new comments

• Check and warn if soft delete policy is enabled for temporary bucket

  #31783 commented on Sep 15, 2024 • 0 new comments

• Poll for backlog in background thread instead of inline

  #31697 commented on Sep 19, 2024 • 0 new comments

• Initial Commit for AvroPayloadSerializer

  #22348 commented on Sep 17, 2024 • 0 new comments

• [Task]: Use "Distroless" Container Images to reduce potential vulnerabilities for Beam Python SDK containers

  #28991 commented on Sep 21, 2024 • 0 new comments

• [Tracking Umbrella] Prism Runner areas for contribution.

  #29650 commented on Sep 21, 2024 • 0 new comments

• Dataflow runner does not respect ParDo's lifecycle on case of exceptions

  #18592 commented on Sep 20, 2024 • 0 new comments

• Clickhouse test failures

  #20912 commented on Sep 19, 2024 • 0 new comments

• Populate labels and custom metadata on VMs that serves Dataflow

  #19453 commented on Sep 18, 2024 • 0 new comments

• The PostCommit XVR GoUsingJava Dataflow job is flaky

  #30519 commented on Sep 17, 2024 • 0 new comments

• [Bug]: cloudpickle appears to incorrectly unpickle cloned combiners

  #26209 commented on Sep 16, 2024 • 0 new comments

• [Tracking]: Performance issue of PubSubIO on non-Dataflow runner

  #31510 commented on Sep 16, 2024 • 0 new comments