Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Spark 4.0.0 Build Profile and Other Supporting Changes [databricks] #10994

Merged
merged 54 commits into from
Jul 16, 2024
Merged

Add Spark 4.0.0 Build Profile and Other Supporting Changes [databricks] #10994

merged 54 commits into from
Jul 16, 2024

Commits on Jun 7, 2024

  1. POM changes for Spark 4.0.0 

    Signed-off-by: Raza Jafri <rjafri@nvidia.com>
    @razajafri
    razajafri committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    0c3a1ba View commit details
    Browse the repository at this point in the history

  2. validate buildver and scala versions

    @razajafri
    razajafri committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    1631ab4 View commit details
    Browse the repository at this point in the history

  3. more pom changes

    @razajafri
    razajafri committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    3271bfd View commit details
    Browse the repository at this point in the history

  4. fixed the scala-2.12 comment

    @razajafri
    razajafri committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    5d2b867 View commit details
    Browse the repository at this point in the history

  5. more fixes for scala-2.13 pom

    @razajafri
    razajafri committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    58806bd View commit details
    Browse the repository at this point in the history

  6. addressed comments

    @razajafri
    razajafri committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    540c732 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. add in shim check to account for 400

    @razajafri
    razajafri committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    5faecf8 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. add 400 for premerge tests against jdk 17

    @razajafri
    razajafri committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    5de0fff View commit details
    Browse the repository at this point in the history

  2. temporarily remove 400 from snapshotScala213

    @razajafri
    razajafri committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    bb784f0 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/branch-24.08' into HEAD

    @razajafri
    razajafri committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    140eb7b View commit details
    Browse the repository at this point in the history

  4. fixed 2.13 pom

    @razajafri
    razajafri committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    41e1982 View commit details
    Browse the repository at this point in the history

  5. Remove 400 from jdk17 as it will compile with Scala 2.12

    @razajafri
    razajafri committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    85a2f6d View commit details
    Browse the repository at this point in the history

  6. github workflow changes

    @razajafri
    razajafri committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    f7f5a98 View commit details
    Browse the repository at this point in the history

  7. added quotes to pom-directory

    @razajafri
    razajafri committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    74dd568 View commit details
    Browse the repository at this point in the history

  8. update version defs to include scala 213 jdk 17

    @razajafri
    razajafri committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    bd1bc70 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Merge remote-tracking branch 'origin/branch-24.08' into HEAD

    @razajafri
    razajafri committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    46eb751 View commit details
    Browse the repository at this point in the history

  2. Cross-compile all shims from JDK17 to JDK8 

    Eliminate Logging inheritance to prevent shimming of unshimmable API
classes

Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    2b15ab2 View commit details
    Browse the repository at this point in the history

  3. dummy

    @gerashegalov
    gerashegalov committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    f7f8edf View commit details
    Browse the repository at this point in the history

  4. undo api pom change 

    Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    ac0ecba View commit details
    Browse the repository at this point in the history

  5. Add preview1 to the allowed shim versions 

    Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    c644ce8 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. Scala 2.13 to require JDK17 

    Signed-off-by: Gera Shegalov <gera@apache.org>
    @gerashegalov
    gerashegalov committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    9d182b3 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #3 from gerashegalov/spark400crosscompile 

     Cross-compile all shims from JDK17 to JDK8
    @razajafri
    razajafri authored Jun 28, 2024
    Configuration menu
    Copy the full SHA
    96e0843 View commit details
    Browse the repository at this point in the history

  3. Removed unused import left over from #3

    @razajafri
    razajafri committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    b51c08b View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2024

  1. Setup JAVA_HOME before caching

    @razajafri
    razajafri committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    1b9beb5 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. Only upgrade the Scala plugin for Scala 2.13

    @razajafri
    razajafri committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    a173f35 View commit details
    Browse the repository at this point in the history

  2. Regenerate Scala 2.13 poms

    @razajafri
    razajafri committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    6138cc8 View commit details
    Browse the repository at this point in the history

  3. Remove 330 from JDK17 builds for Scala 2.12

    @razajafri
    razajafri committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    1faabd4 View commit details
    Browse the repository at this point in the history

  4. Revert "Remove 330 from JDK17 builds for Scala 2.12" 

    This reverts commit 1faabd4.
    @razajafri
    razajafri committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    0cf0036 View commit details
    Browse the repository at this point in the history

  5. Downgrade scala.plugin.version for cloudera

    @razajafri
    razajafri committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    a7b42c6 View commit details
    Browse the repository at this point in the history

  6. Updated comment to include the issue

    @razajafri
    razajafri committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    8d0f8ca View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. Upgrading the scala.maven.plugin version to 4.9.1 which is the same a… 

    …s Spark 4.0.0
    @razajafri
    razajafri committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    45b0d57 View commit details
    Browse the repository at this point in the history

  2. Downgrade scala-maven-plugin for Cloudera

    @razajafri
    razajafri committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    eb09f98 View commit details
    Browse the repository at this point in the history

  3. revert mvn verify changes

    @razajafri
    razajafri committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    4407dbf View commit details
    Browse the repository at this point in the history

  4. Avoid cache for JDK 17 

    * removed cache dep from scala 213
* Added Scala 2.13 specific checks
    @razajafri
    razajafri committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    0e4e45a View commit details
    Browse the repository at this point in the history

  5. Handle the change for UnaryPositive now extending RuntimeReplaceable

    @razajafri
    razajafri committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    bd10267 View commit details
    Browse the repository at this point in the history

  6. Removing 330 from jdk17.buildvers as we only support Scala2.13 and fi… 

    …xing the enviornment variable in version-defs.sh that we read for building against JDK17 with Scala 213
    @razajafri
    razajafri committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    2982a59 View commit details
    Browse the repository at this point in the history

  7. Update Scala 2.13 poms

    @razajafri
    razajafri committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    319aefc View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. Merge remote-tracking branch 'origin/branch-24.08' into HEAD

    @razajafri
    razajafri committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    835fbb4 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. fixed scala2.13 verify to actually use the scala2.13/pom.xml

    @razajafri
    razajafri committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    dfbb149 View commit details
    Browse the repository at this point in the history

  2. Added missing csv files

    @razajafri
    razajafri committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    8ffc4f1 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. Merge remote-tracking branch 'origin/branch-24.08' into HEAD

    @razajafri
    razajafri committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    f599413 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. Skip Opcode tests 

    There is a bytecode incompatibility which is why we are skipping these
until we add support for it. For details please see the following two
issues
NVIDIA#11174
NVIDIA#10203
    @razajafri
    razajafri committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    0544c6f View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/branch-24.08' into HEAD

    @razajafri
    razajafri committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    a43a68d View commit details
    Browse the repository at this point in the history

  3. upmerged and fixed the new compile error introduced

    @razajafri
    razajafri committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    2cf7351 View commit details
    Browse the repository at this point in the history

  4. addressed review comments

    @razajafri
    razajafri committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    cabcda0 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/branch-24.08' into HEAD

    @razajafri
    razajafri committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    c19fccf View commit details
    Browse the repository at this point in the history

  6. Removed jdk17 cloudera check and moved it inside the 321,330 and 332 … 

    …cloudera profiles
    @razajafri
    razajafri committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    feeabfc View commit details
    Browse the repository at this point in the history

  7. fixed upmerge conflicts

    @razajafri
    razajafri committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    54a3ee4 View commit details
    Browse the repository at this point in the history

  8. reverted renaming of id

    @razajafri
    razajafri committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    48fe626 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. Merge remote-tracking branch 'origin/branch-24.08' into SP-9259-POM-c… 

    …hanges
    @razajafri
    razajafri committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    166e4c6 View commit details
    Browse the repository at this point in the history

  2. Fixed HiveGenericUDFShim

    @razajafri
    razajafri committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6cbce78 View commit details
    Browse the repository at this point in the history

  3. addressed review comments

    @razajafri
    razajafri committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8d0351b View commit details
    Browse the repository at this point in the history

  4. reverted the debugging code

    @razajafri
    razajafri committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    5b0e36d View commit details
    Browse the repository at this point in the history

  5. generated Scala 2.13 poms

    @razajafri
    razajafri committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    66419b8 View commit details
    Browse the repository at this point in the history