Publishing website 2024/05/02 23:37:30 at commit 6fdabea

website/generated-content/blog/beam-2.55.0/index.html

@@ -40,7 +40,7 @@
 This release includes both improvements and new functionality.
 See the <a href=/get-started/downloads/#2550-2023-03-25>download page</a> for this release.</p><p>For more information on changes in 2.55.0, check out the <a href=https://github.com/apache/beam/milestone/19>detailed release notes</a>.</p><h2 id=highlights>Highlights</h2><ul><li>The Python SDK will now include automatically generated wrappers for external Java transforms! (<a href=https://github.com/apache/beam/pull/29834>#29834</a>)</li></ul><h2 id=ios>I/Os</h2><ul><li>Added support for handling bad records to BigQueryIO (<a href=https://github.com/apache/beam/pull/30081>#30081</a>).<ul><li>Full Support for Storage Read and Write APIs</li><li>Partial Support for File Loads (Failures writing to files supported, failures loading files to BQ unsupported)</li><li>No Support for Extract or Streaming Inserts</li></ul></li><li>Added support for handling bad records to PubSubIO (<a href=https://github.com/apache/beam/pull/30372>#30372</a>).<ul><li>Support is not available for handling schema mismatches, and enabling error handling for writing to Pub/Sub topics with schemas is not recommended</li></ul></li><li><code>--enableBundling</code> pipeline option for BigQueryIO DIRECT_READ is replaced by <code>--enableStorageReadApiV2</code>. Both were considered experimental and subject to change (Java) (<a href=https://github.com/apache/beam/issues/26354>#26354</a>).</li></ul><h2 id=new-features--improvements>New Features / Improvements</h2><ul><li>Allow writing clustered and not time-partitioned BigQuery tables (Java) (<a href=https://github.com/apache/beam/pull/30094>#30094</a>).</li><li>Redis cache support added to RequestResponseIO and Enrichment transform (Python) (<a href=https://github.com/apache/beam/pull/30307>#30307</a>)</li><li>Merged <code>sdks/java/fn-execution</code> and <code>runners/core-construction-java</code> into the main SDK. These artifacts were never meant for users, but noting
 that they no longer exist. These are steps to bring portability into the core SDK alongside all other core functionality.</li><li>Added Vertex AI Feature Store handler for Enrichment transform (Python) (<a href=https://github.com/apache/beam/pull/30388>#30388</a>)</li></ul><h2 id=breaking-changes>Breaking Changes</h2><ul><li>Arrow version was bumped to 15.0.0 from 5.0.0 (<a href=https://github.com/apache/beam/pull/30181>#30181</a>).</li><li>Go SDK users who build custom worker containers may run into issues with the move to distroless containers as a base (see Security Fixes).<ul><li>The issue stems from distroless containers lacking additional tools, which current custom container processes may rely on.</li><li>See <a href=https://beam.apache.org/documentation/runtime/environments/#from-scratch-go>https://beam.apache.org/documentation/runtime/environments/#from-scratch-go</a> for instructions on building and using a custom container.</li></ul></li><li>Python SDK has changed the default value for the <code>--max_cache_memory_usage_mb</code> pipeline option from 100 to 0. This option was first introduced in the 2.52.0 SDK version. This change restores the behavior of the 2.51.0 SDK, which does not use the state cache. If your pipeline uses iterable side inputs views, consider increasing the cache size by setting the option manually. (<a href=https://github.com/apache/beam/issues/30360>#30360</a>).</li></ul><h2 id=deprecations>Deprecations</h2><ul><li>N/A</li></ul><h2 id=bug-fixes>Bug fixes</h2><ul><li>Fixed <code>SpannerIO.readChangeStream</code> to support propagating credentials from pipeline options
-to the <code>getDialect</code> calls for authenticating with Spanner (Java) (<a href=https://github.com/apache/beam/pull/30361>#30361</a>).</li><li>Reduced the number of HTTP requests in GCSIO function calls (Python) (<a href=https://github.com/apache/beam/pull/30205>#30205</a>)</li></ul><h2 id=security-fixes>Security Fixes</h2><ul><li>Go SDK base container image moved to distroless/base-nossl-debian12, reducing vulnerable container surface to kernel and glibc (<a href=https://github.com/apache/beam/pull/30011>#30011</a>).</li></ul><h2 id=known-issues>Known Issues</h2><ul><li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href=https://github.com/apache/beam/pull/30679>#30679</a>).</li></ul><h2 id=list-of-contributors>List of Contributors</h2><p>According to git shortlog, the following people contributed to the {$RELEASE_VERSION} release. Thank you to all contributors!</p><p>Ahmed Abualsaud</p><p>Anand Inguva</p><p>Andrew Crites</p><p>Andrey Devyatkin</p><p>Arun Pandian</p><p>Arvind Ram</p><p>Chamikara Jayalath</p><p>Chris Gray</p><p>Claire McGinty</p><p>Damon Douglas</p><p>Dan Ellis</p><p>Danny McCormick</p><p>Daria Bezkorovaina</p><p>Dima I</p><p>Edward Cui</p><p>Ferran Fernández Garrido</p><p>GStravinsky</p><p>Jan Lukavský</p><p>Jason Mitchell</p><p>JayajP</p><p>Jeff Kinard</p><p>Jeffrey Kinard</p><p>Kenneth Knowles</p><p>Mattie Fu</p><p>Michel Davit</p><p>Oleh Borysevych</p><p>Ritesh Ghorse</p><p>Ritesh Tarway</p><p>Robert Bradshaw</p><p>Robert Burke</p><p>Sam Whittle</p><p>Scott Strong</p><p>Shunping Huang</p><p>Steven van Rossum</p><p>Svetak Sundhar</p><p>Talat UYARER</p><p>Ukjae Jeong (Jay)</p><p>Vitaly Terentyev</p><p>Vlado Djerek</p><p>Yi Hu</p><p>akashorabek</p><p>case-k</p><p>clmccart</p><p>dengwe1</p><p>dhruvdua</p><p>hardshah</p><p>johnjcasey</p><p>liferoad</p><p>martin trieu</p><p>tvalentyn</p></div></div><div class=blog-content><h2>Latest from the blog</h2></div><div class=posts-list><a class=post-card href=/blog/beam-yaml-release/ data-categories="blog "><div class="post-info post-category"><p>blog</p><p>2024/04/11</p></div><div class=post><p class=post-title>Introducing Beam YAML: Apache Beam's First No-code SDK</p><p class=post-info>Jeff Kinard</p></div></a><a class=post-card href=/blog/beam-2.55.0/ data-categories="blog release "><div class="post-info post-category"><p>blog & release
+to the <code>getDialect</code> calls for authenticating with Spanner (Java) (<a href=https://github.com/apache/beam/pull/30361>#30361</a>).</li><li>Reduced the number of HTTP requests in GCSIO function calls (Python) (<a href=https://github.com/apache/beam/pull/30205>#30205</a>)</li></ul><h2 id=security-fixes>Security Fixes</h2><ul><li>Go SDK base container image moved to distroless/base-nossl-debian12, reducing vulnerable container surface to kernel and glibc (<a href=https://github.com/apache/beam/pull/30011>#30011</a>).</li></ul><h2 id=known-issues>Known Issues</h2><ul><li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href=https://github.com/apache/beam/pull/30679>#30679</a>).</li></ul><h2 id=list-of-contributors>List of Contributors</h2><p>According to git shortlog, the following people contributed to the 2.55.0 release. Thank you to all contributors!</p><p>Ahmed Abualsaud</p><p>Anand Inguva</p><p>Andrew Crites</p><p>Andrey Devyatkin</p><p>Arun Pandian</p><p>Arvind Ram</p><p>Chamikara Jayalath</p><p>Chris Gray</p><p>Claire McGinty</p><p>Damon Douglas</p><p>Dan Ellis</p><p>Danny McCormick</p><p>Daria Bezkorovaina</p><p>Dima I</p><p>Edward Cui</p><p>Ferran Fernández Garrido</p><p>GStravinsky</p><p>Jan Lukavský</p><p>Jason Mitchell</p><p>JayajP</p><p>Jeff Kinard</p><p>Jeffrey Kinard</p><p>Kenneth Knowles</p><p>Mattie Fu</p><p>Michel Davit</p><p>Oleh Borysevych</p><p>Ritesh Ghorse</p><p>Ritesh Tarway</p><p>Robert Bradshaw</p><p>Robert Burke</p><p>Sam Whittle</p><p>Scott Strong</p><p>Shunping Huang</p><p>Steven van Rossum</p><p>Svetak Sundhar</p><p>Talat UYARER</p><p>Ukjae Jeong (Jay)</p><p>Vitaly Terentyev</p><p>Vlado Djerek</p><p>Yi Hu</p><p>akashorabek</p><p>case-k</p><p>clmccart</p><p>dengwe1</p><p>dhruvdua</p><p>hardshah</p><p>johnjcasey</p><p>liferoad</p><p>martin trieu</p><p>tvalentyn</p></div></div><div class=blog-content><h2>Latest from the blog</h2></div><div class=posts-list><a class=post-card href=/blog/beam-yaml-release/ data-categories="blog "><div class="post-info post-category"><p>blog</p><p>2024/04/11</p></div><div class=post><p class=post-title>Introducing Beam YAML: Apache Beam's First No-code SDK</p><p class=post-info>Jeff Kinard</p></div></a><a class=post-card href=/blog/beam-2.55.0/ data-categories="blog release "><div class="post-info post-category"><p>blog & release
 
    </p><p>2024/03/25</p></div><div class=post><p class=post-title>Apache Beam 2.55.0</p><p class=post-info>Yi Hu</p></div></a><a class=post-card href=/blog/beam-2.54.0/ data-categories="blog release "><div class="post-info post-category"><p>blog & release
 

website/generated-content/blog/beam-2.56.0/index.html

@@ -39,7 +39,7 @@
        </p><p>2024/05/01</p></div><header class=post-header><h2 itemprop="name headline">Apache Beam 2.56.0</h1><div class=post-info><span>Danny McCormick</span></div></header><div class="arrow-list header-top-margin" itemprop=articleBody><p>We are happy to present the new 2.56.0 release of Beam.
 This release includes both improvements and new functionality.
 See the <a href=/get-started/downloads/#2550-2023-03-25>download page</a> for this release.</p><p>For more information on changes in 2.56.0, check out the <a href=https://github.com/apache/beam/milestone/20>detailed release notes</a>.</p><h2 id=highlights>Highlights</h2><ul><li>Added FlinkRunner for Flink 1.17, removed support for Flink 1.12 and 1.13. Previous version of Pipeline running on Flink 1.16 and below can be upgraded to 1.17, if the Pipeline is first updated to Beam 2.56.0 with the same Flink version. After Pipeline runs with Beam 2.56.0, it should be possible to upgrade to FlinkRunner with Flink 1.17. (<a href=https://github.com/apache/beam/issues/29939>#29939</a>)</li></ul><h2 id=ios>I/Os</h2><ul><li>Upgraded Avro version to 1.11.3, kafka-avro-serializer and kafka-schema-registry-client versions to 7.6.0 (Java) (<a href=https://github.com/apache/beam/pull/30638>#30638</a>).
-The newer Avro package is known to have breaking changes. If you are affected, you can keep pinned to older Avro versions which are also tested with Beam.</li></ul><h2 id=new-features--improvements>New Features / Improvements</h2><ul><li>Profiling of Cythonized code has been disabled by default. This might improve performance for some Python pipelines (<a href=https://github.com/apache/beam/pull/30938>#30938</a>).</li><li>Bigtable enrichment handler now accepts a custom function to build a composite row key. (Python) (<a href=https://github.com/apache/beam/issues/30975>#30974</a>).</li></ul><h2 id=breaking-changes>Breaking Changes</h2><ul><li>Default consumer polling timeout for KafkaIO.Read was increased from 1 second to 2 seconds. Use KafkaIO.read().withConsumerPollingTimeout(Duration duration) to configure this timeout value when necessary (<a href=https://github.com/apache/beam/issues/30870>#30870</a>).</li><li>Python Dataflow users no longer need to manually specify &ndash;streaming for pipelines using unbounded sources such as ReadFromPubSub.</li></ul><h2 id=bugfixes>Bugfixes</h2><ul><li>Fixed locking issue when shutting down inactive bundle processors. Symptoms of this issue include slowness or stuckness in long-running jobs (Python) (<a href=https://github.com/apache/beam/pull/30679>#30679</a>).</li><li>Fixed logging issue that caused silecing the pip output when installing of dependencies provided in <code>--requirements_file</code> (Python).</li></ul><h2 id=list-of-contributors>List of Contributors</h2><p>According to git shortlog, the following people contributed to the {$RELEASE_VERSION} release. Thank you to all contributors!</p><p>Abacn</p><p>Ahmed Abualsaud</p><p>Andrei Gurau</p><p>Andrey Devyatkin</p><p>Aravind Pedapudi</p><p>Arun Pandian</p><p>Arvind Ram</p><p>Bartosz Zablocki</p><p>Brachi Packter</p><p>Byron Ellis</p><p>Chamikara Jayalath</p><p>Clement DAL PALU</p><p>Damon</p><p>Danny McCormick</p><p>Daria Bezkorovaina</p><p>Dip Patel</p><p>Evan Burrell</p><p>Hai Joey Tran</p><p>Jack McCluskey</p><p>Jan Lukavský</p><p>JayajP</p><p>Jeff Kinard</p><p>Julien Tournay</p><p>Kenneth Knowles</p><p>Luís Bianchin</p><p>Maciej Szwaja</p><p>Melody Shen</p><p>Oleh Borysevych</p><p>Pablo Estrada</p><p>Rebecca Szper</p><p>Ritesh Ghorse</p><p>Robert Bradshaw</p><p>Sam Whittle</p><p>Sergei Lilichenko</p><p>Shahar Epstein</p><p>Shunping Huang</p><p>Svetak Sundhar</p><p>Timothy Itodo</p><p>Veronica Wasson</p><p>Vitaly Terentyev</p><p>Vlado Djerek</p><p>Yi Hu</p><p>akashorabek</p><p>bzablocki</p><p>clmccart</p><p>damccorm</p><p>dependabot[bot]</p><p>dmitryor</p><p>github-actions[bot]</p><p>liferoad</p><p>martin trieu</p><p>tvalentyn</p><p>xianhualiu</p></div></div><div class=blog-content><h2>Latest from the blog</h2></div><div class=posts-list><a class=post-card href=/blog/beam-yaml-release/ data-categories="blog "><div class="post-info post-category"><p>blog</p><p>2024/04/11</p></div><div class=post><p class=post-title>Introducing Beam YAML: Apache Beam's First No-code SDK</p><p class=post-info>Jeff Kinard</p></div></a><a class=post-card href=/blog/beam-2.55.0/ data-categories="blog release "><div class="post-info post-category"><p>blog & release
+The newer Avro package is known to have breaking changes. If you are affected, you can keep pinned to older Avro versions which are also tested with Beam.</li></ul><h2 id=new-features--improvements>New Features / Improvements</h2><ul><li>Profiling of Cythonized code has been disabled by default. This might improve performance for some Python pipelines (<a href=https://github.com/apache/beam/pull/30938>#30938</a>).</li><li>Bigtable enrichment handler now accepts a custom function to build a composite row key. (Python) (<a href=https://github.com/apache/beam/issues/30975>#30974</a>).</li></ul><h2 id=breaking-changes>Breaking Changes</h2><ul><li>Default consumer polling timeout for KafkaIO.Read was increased from 1 second to 2 seconds. Use KafkaIO.read().withConsumerPollingTimeout(Duration duration) to configure this timeout value when necessary (<a href=https://github.com/apache/beam/issues/30870>#30870</a>).</li><li>Python Dataflow users no longer need to manually specify &ndash;streaming for pipelines using unbounded sources such as ReadFromPubSub.</li></ul><h2 id=bugfixes>Bugfixes</h2><ul><li>Fixed locking issue when shutting down inactive bundle processors. Symptoms of this issue include slowness or stuckness in long-running jobs (Python) (<a href=https://github.com/apache/beam/pull/30679>#30679</a>).</li><li>Fixed logging issue that caused silecing the pip output when installing of dependencies provided in <code>--requirements_file</code> (Python).</li></ul><h2 id=list-of-contributors>List of Contributors</h2><p>According to git shortlog, the following people contributed to the 2.56.0 release. Thank you to all contributors!</p><p>Abacn</p><p>Ahmed Abualsaud</p><p>Andrei Gurau</p><p>Andrey Devyatkin</p><p>Aravind Pedapudi</p><p>Arun Pandian</p><p>Arvind Ram</p><p>Bartosz Zablocki</p><p>Brachi Packter</p><p>Byron Ellis</p><p>Chamikara Jayalath</p><p>Clement DAL PALU</p><p>Damon</p><p>Danny McCormick</p><p>Daria Bezkorovaina</p><p>Dip Patel</p><p>Evan Burrell</p><p>Hai Joey Tran</p><p>Jack McCluskey</p><p>Jan Lukavský</p><p>JayajP</p><p>Jeff Kinard</p><p>Julien Tournay</p><p>Kenneth Knowles</p><p>Luís Bianchin</p><p>Maciej Szwaja</p><p>Melody Shen</p><p>Oleh Borysevych</p><p>Pablo Estrada</p><p>Rebecca Szper</p><p>Ritesh Ghorse</p><p>Robert Bradshaw</p><p>Sam Whittle</p><p>Sergei Lilichenko</p><p>Shahar Epstein</p><p>Shunping Huang</p><p>Svetak Sundhar</p><p>Timothy Itodo</p><p>Veronica Wasson</p><p>Vitaly Terentyev</p><p>Vlado Djerek</p><p>Yi Hu</p><p>akashorabek</p><p>bzablocki</p><p>clmccart</p><p>damccorm</p><p>dependabot[bot]</p><p>dmitryor</p><p>github-actions[bot]</p><p>liferoad</p><p>martin trieu</p><p>tvalentyn</p><p>xianhualiu</p></div></div><div class=blog-content><h2>Latest from the blog</h2></div><div class=posts-list><a class=post-card href=/blog/beam-yaml-release/ data-categories="blog "><div class="post-info post-category"><p>blog</p><p>2024/04/11</p></div><div class=post><p class=post-title>Introducing Beam YAML: Apache Beam's First No-code SDK</p><p class=post-info>Jeff Kinard</p></div></a><a class=post-card href=/blog/beam-2.55.0/ data-categories="blog release "><div class="post-info post-category"><p>blog & release
 
    </p><p>2024/03/25</p></div><div class=post><p class=post-title>Apache Beam 2.55.0</p><p class=post-info>Yi Hu</p></div></a><a class=post-card href=/blog/beam-2.54.0/ data-categories="blog release "><div class="post-info post-category"><p>blog & release
 

website/generated-content/blog/index.xml

@@ -39,7 +39,7 @@ The newer Avro package is known to have breaking changes. If you are affected, y
 <li>Fixed logging issue that caused silecing the pip output when installing of dependencies provided in <code>--requirements_file</code> (Python).</li>
 </ul>
 <h2 id="list-of-contributors">List of Contributors</h2>
-<p>According to git shortlog, the following people contributed to the {$RELEASE_VERSION} release. Thank you to all contributors!</p>
+<p>According to git shortlog, the following people contributed to the 2.56.0 release. Thank you to all contributors!</p>
 <p>Abacn</p>
 <p>Ahmed Abualsaud</p>
 <p>Andrei Gurau</p>
@@ -397,7 +397,7 @@ to the <code>getDialect</code> calls for authenticating with Spanner (Java
 <li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li>
 </ul>
 <h2 id="list-of-contributors">List of Contributors</h2>
-<p>According to git shortlog, the following people contributed to the {$RELEASE_VERSION} release. Thank you to all contributors!</p>
+<p>According to git shortlog, the following people contributed to the 2.55.0 release. Thank you to all contributors!</p>
 <p>Ahmed Abualsaud</p>
 <p>Anand Inguva</p>
 <p>Andrew Crites</p>

website/generated-content/categories/blog/index.xml

@@ -39,7 +39,7 @@ The newer Avro package is known to have breaking changes. If you are affected, y
 <li>Fixed logging issue that caused silecing the pip output when installing of dependencies provided in <code>--requirements_file</code> (Python).</li>
 </ul>
 <h2 id="list-of-contributors">List of Contributors</h2>
-<p>According to git shortlog, the following people contributed to the {$RELEASE_VERSION} release. Thank you to all contributors!</p>
+<p>According to git shortlog, the following people contributed to the 2.56.0 release. Thank you to all contributors!</p>
 <p>Abacn</p>
 <p>Ahmed Abualsaud</p>
 <p>Andrei Gurau</p>
@@ -397,7 +397,7 @@ to the <code>getDialect</code> calls for authenticating with Spanner (Java
 <li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li>
 </ul>
 <h2 id="list-of-contributors">List of Contributors</h2>
-<p>According to git shortlog, the following people contributed to the {$RELEASE_VERSION} release. Thank you to all contributors!</p>
+<p>According to git shortlog, the following people contributed to the 2.55.0 release. Thank you to all contributors!</p>
 <p>Ahmed Abualsaud</p>
 <p>Anand Inguva</p>
 <p>Andrew Crites</p>