Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MariaDB container volume mounts are incorrect #5877

Closed
corneil opened this issue Jul 30, 2024 · 2 comments · Fixed by #5875
Closed

MariaDB container volume mounts are incorrect #5877

corneil opened this issue Jul 30, 2024 · 2 comments · Fixed by #5875
Assignees

Comments

@corneil
Copy link
Contributor

corneil commented Jul 30, 2024

Description:
The volume mount for mariadb deployment was created with /var/lib/mariadb and results in a blank database after a restart because the mount path is expected to be /var/lib/mysql

Release versions:
2.8.x

Fix
The correct mount path is /var/lib/mysql

@corneil corneil self-assigned this Jul 30, 2024
corneil added a commit to corneil/spring-cloud-dataflow that referenced this issue Jul 30, 2024
@cppwfs
Copy link
Contributor

cppwfs commented Jul 30, 2024

I understand the need for the fix, in that if a user shuts down SCDF or Skipper the streams definitions page will be blank.
However using the instructions here I am unable to reproduce the base case. What setup would exhibit the behavior. Thanks!

@corneil
Copy link
Contributor Author

corneil commented Jul 31, 2024

Do the following:

source ./src/deploy/k8s/use-mk-docker.sh mariadb rabbit --snapshot
./src/deploy/k8s/configure-k8s.sh
# minikube tunnel # in separate terminal
./src/deploy/k8s/load-image.sh mariadb:10.6
export NS=scdf
kubectl create namespace "$NS"
kubectl create secret docker-registry registry-key --namespace "$NS" --docker-server=$DOCKER_SERVER --docker-username=$DOCKER_USER --docker-password=$DOCKER_PASSWORD --docker-email=$DOCKER_EMAIL
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "registry-key"}]}' --namespace "$NS"
kubectl create --namespace $NS -f ./src/kubernetes/mariadb/
kubectl rollout status deployment --namespace "$NS" mariadb

Use k9s to forward mariadb port.
Use IntelliJ Database tools and configure connection to local MariaDB with root/yourpassword

Execute src/main/resources/schemas/mariadb/V1-dataflow.sql

Then execute:

minikube stop
minikube start

Use k9s to forward port on mariadb again.
Check connection and tables in IntelliJ database tools.

corneil added a commit that referenced this issue Aug 1, 2024
cppwfs pushed a commit to cppwfs/spring-cloud-dataflow that referenced this issue Aug 23, 2024
…ng-cloud-deployer#465 [skip ci]

Update paragraph for multiple Init Containers.

Fix mariadb mount to be /var/lib/mysql (spring-cloud#5875)

Fixes spring-cloud#5877

Add spaces after columns for clarity.  This is done during merge
cppwfs pushed a commit to cppwfs/spring-cloud-dataflow that referenced this issue Aug 23, 2024
…ng-cloud-deployer#465 [skip ci]

Update paragraph for multiple Init Containers.

Fix mariadb mount to be /var/lib/mysql (spring-cloud#5875)

Fixes spring-cloud#5877

Add spaces after columns for clarity.  This is done during merge
corneil added a commit that referenced this issue Oct 2, 2024
* Update build files for main-3 [skip-ci]

* Update SCDF to use Boot 3.2.2

* Jakarta updates
* Updated for Logging Date Format
* Replace spring.factories with autoconfig imports
* Constructor Bindings removed
* Readd the yaml files that caused the migrator to fail
* Update SCDF to Boot 3.2.2 as parent
* Updated dependencies where needed
* Added versions to some dependencies where versions are no longer in the bom
* Updated hibernate usages where class names changed
* Got to spring-cloud-common-security-config-web.
** The OauthSecurityCOnfiguration has @ConditionalOnMissingBean for the WebSecurityAdapter.
** Which is no longer available.  We need to determine what should go here.
* Update workflows to use Java 17

* Make SCDF and Skipper compilable for boot 3.x

* Replaced SocketUtils with TestSocketUtils
** It was moved to the test package.
* Migrated httpclient to httpclient5
** Removed use of httpclient 4.x dependencies added yesterday
* Updated Types in AuditRecord Entity to use JdbcTypeCode
* Security Modules need to be compiled with Boot3 and securty 6
* Update SCDF to use deployer 3.0.x
* Update SCDF pom files to create Java 17 jars
* Exclude javax.annotation from deployer artifacts.
* Add jakarta annotation dependencies
* Update code from javax to jakarta
* Replaced Entity  @type with  @JavaTypeCode
* Update JobParam to batch 5.
* Update httpclient package to httpclient5 package

* Update SCDF so that it can be compiled with Boot 3.2.x.  Part 3

Added dataflow specific PageQueryProvider for missing functions

Batch removed some methods that are still required by dataflow.
Created dataflow version of those classes so that we can implement those methods

Added hibernate version

Checkpoint for server core and server

Updated Skipper components

Update classic docs to be restful doc 3.0 compliant

Update code to resolve review comments

Thank you @cbono!!!!

One last polish

* Remove dependency overrides from 2.11.x (Boot 2.x)

This commit removes the following dependency overrides that were
put in place to override earlier dependencies in Boot 2.x:
- logback
- jackson
- snakeyaml

Also, the joda-time version is updated to `2.12.7`

* Remove duplicated maven-source-plugin from `spring-cloud-dataflow-common/**`

Update Awaitility and Semver libs

This commit removes the dependency management for `com.jayway.awaitility` in
favor of the Spring Boot managed version at the new `org.awaitility`
coordinates.

Also, the Semver lib is updated to 0.10.2

* Remove <dependencyManagement> of `testcontainers-bom`

Dependency management of `testcontainers-bom` is no longer needed as
Spring Boot 3.x provides the version for the `testcontainers-bom`.

* Remove remaining dependency override of snakeyaml.

A previous commit removed the dependency overrides for snakeyaml in
`spring-cloud-dataflow-parent` (which ~50% of the modules extend from).
This commit removes the dependency overrides for snakeyaml in
`spring-cloud-dataflow-build-dependencies` (the remaining 50% of modules extend
from this).

* Remove <dependencyManagement> of `junit-jupiter-bom`

Dependency management of `junit-jupiter-bom` is no longer needed as
Spring Boot 3.x provides the version for the `junit-jupiter-bom`.

* Remove dependency override of json-smart.

The version was previously overridden to `2.4.11` due to CVE.
This is no longer needed as Spring Boot provides the version of `2.5.0`.

* Remove <dependencyManagement> of `postgresql`

Dependency management of `postgresql` is not needed as
Spring Boot 3.x provides the version for `postgresql`.

* Remove unused `spring.version` property

* Update DockerCompose tests to Junit5

Also had to account for some changes in Mockito w/ varargs methods.

Suggested changes from PR review

* Use AutoConfiguration.imports for all auto-configs

Fix from PR review

* Add back missing nested @configuration annotations

* Fix startup and EPP ordering (#5670)

* Fix tests for rest-resource (SCDF3 migration)

The new HTTP client requires access to the host:port. As such, we now use
@SpringBootTest to launch web server to allow test to proceed.

Also include H2 test dependency to deal w/ test startup failure.

* Fix tests for rest-client (SCDF3 migration)

This commit excludes DataSourceAutoConfiguration from the tests.

* Update SpringDoc OpenAPI to latest 2.x

This commit updates to Open API 2.x which is the version that supports Boot 3.

The SpringDocs migration guide was followed:
https://springdoc.org/#migrating-from-springdoc-v1

* Change vscode settings

Add recommendation from vscode itself not that we're a monorepo
lsp needs more memory.

* Use config legacy processing in Skipper (#5673)

* Refactor TaskLauncherSink to SCSt Consumer

Prior to this commit the sink used a PollableMessageSource with a trigger to
source its inbound messages. This commit replaces that mechanism w/ a simple
SCSt consumer with a @StreamRetryTemplate.

* Add Autoconfiguration.imports to resource includes in pom.xmls

We had an issue where some of the auto configurations were not firing.
This is because when a user specifies the resource tag in maven,
the boot plugin expects the user to fill the resources manually vs. Its default behavior

Also removed the version from the H2 dependency so that it can be managed by the bom

Removed unnecessary @configuration annotaions where a @autoConfiguration annotation is present

* Add metrics exclusions till metric migration is complete

Currently SCDF fails to start with errors around metrics.
These will need to be re-added when metric migration begins

Remove debug settings from previous commit

* Stub out DataflowPagingProvider to allow Dataflow to start (#5679)

* Stub out DataflowPagingProviders to allow Dataflow to start 
* Provide PlatformTransactionManager to JobExplorer since Batch 5 no longer provides it
* Compile with `-parameters` option

* What was done
For each database type the following migration was implemented:

Prefix the TASK V2 and BATCH V4 tables with V2_ .
This allows user to determine what they wish to do with this data
Remove BOOT3_prefix for TASK V2 and BATCH V5 tables
Make sure that these migrations were supported by flyway
Make sure that the Migration SQL scripts were added to associated yaml files.
What was affected
The following databases were migrated:

H2
MariaDB
Mysql
Postgres
Oracle
SQLServer
DB2
Types of migrations
There were 3 types of migrations that occurred.

Default - These types of migrations were typically alter tables and alter sequences.
The following databases belong to this group:
a. MariaDB
b. Mysql
c. Postgres
d. Oracle (with some exceptions)
e. Sql Server (commands different than others, but principles remained).

In-Memory - For this case the Task V2 and Batch V4 tables/sequences DDL
was removed and BOOT3_ prefix was removed. The only in-memory we support is H2.

DB2 - Gets its own category
To rename the tables db2 requires all primary and foreign keys to be dropped.
In this case we decided to create the new tables and copy the contents of the
original tables to the new tables. Then remove the original.
This was to avoid errors when recreating the keys

How to eat this Elephant
This PR is a bit large so let's discuss how we can handle this review.
Let's review this by sampling one database from each type of migration.
This is so that we can make sure the general pattern works for folks.

Default type, look at Mariadb
In Memory type look at H2
DB2 Type look at (well... ) DB2 :-D
After we finish this sampling review, those changes will be applied to the other databases and then a full review
can be made.

* Update Tests to resolve failures due to updated dependency versions (#5680)

This commit does not cover all the cases where updated dependencies broke existing tests.
The main ones resolved here are some basics on Hibernate dialect versions, batch 5 updates,
and some basic removal of BOOT3/BOOT2 additions in 2.11.x

There are other updates that are causing alot of failures.  Here are the issues and a brief description:
* Currently Hibernate 6 does not create sequences out of the box.   More can be read here: https://github.com/hibernate/hibernate-orm/blob/6.0/migration-guide.adoc#implicit-identifier-sequence-and-table-name
One test that shows this error:TabOnTapCompletionProviderTests

* Hibernate Dialect.  Some tests require the hibernate dialects to be updated to hibernate 6. I did resolve the ones I found.   But may have missed others.

* Many tests are failing trying with the following exception: `No bean named 'mvcConversionService' available`
For example: SpringDocAutoConfigurationTests.enabledWithCustomSpringDocSettings

* The TODO excluding wavefront are causing some errors for example: Error creating bean with name 'management.wavefront-org.springframework.boot.actuate.autoconfigure.wavefront.WavefrontProperties': Lookup method resolution failed
An Example can be found here: SpringDocIntegrationTests.disabledByDefault

* Add skipper to vscode launch config

* Use the single hibernate_sequence strategy

Hibernate 6.0 now creates a sequence per entity hierarchy instead of a single
hibernate_sequence. This commit sets the
`hibernate.id.db_structure_naming_strategy` property to `single` to preserve the
previous behavior of using a single hibernate_sequence.

* Fix SpringDocAutoConfigurationTests

This commit adds a mock "mvcConversionService" bean to the app context for the
SpringDocAutoConfigurationTests as SpringDoc 2.x expects this bean to be
available.

This commit also updates the use of Mockito verify for varargs in the
SpringDocAutoConfigurationTests.

* Overhaul metric deps and options

Align wavefront version so that we don't have misaligned versions
coming out from other parts of a metric system. Short story
is that boot doesn't manage wavefront but there is an explicit
dependency to wavefront sdk libs in metric system.

Rename metric options within management for influx, prometheus
and wavefront to align changes in boot itself.

For rsocket proxy keep old management.metrics.export.prometheus.rsocket.enabled
as that is going to get moved under micrometer spesific namespace when
they release boot3 support.

This commit was supposed to get skipper server to start but there
is a new issue about missing bean
org.springframework.statemachine.data.jpa.JpaStateMachineRepository
which will need to get fixed in a separate commit.

Fixes #5675

* Disable all tests that are failling because of schema changes. (#5688)

This commit marks tests that are failing due to schema changes w/ the 
`TODO: Boot3x followup` so that they can be re-enabled in the future 
once we update the code to handle the schema changes

* Remove trailing slashes from controller tests (#5693)

This commit accounts for the fact that as of Spring Framework 6.0, the trailing 
slash matching configuration option has been deprecated and its default value 
set to false. The mvc tests are adjusted to remove the trailing slash from their
endpoint urls.

* Add DatabaseAwareLobUserType for @lob columns

Hibernate 6.0 removed support for string values in @type mappings in favor of
specifying UserTypes. This commit creates a DatabaseAwareLobUserType that
provides the same functionality as the previous DatabaseAwareLobType.

Add header+copyright

* Fix DefaultPackageReader for Snakeyaml 2.0

This commit allows the PackageMetadata.class to be loaded
with Snakeyaml 2.0

* Remove unused spring-boot-loader dependency

This commit removes the dependency on the newer spring-boot-loader from the
spring-cloud-dataflow-container-registry module. With Spring Boot 3.x, you must
choose only one of these loader options. Otherwise, duplicate non-compatible
classes may be loaded - for example the ZipInflaterInputStream.

* Added DockerHub login to CI to prevent failure of TestContainer tests. (#5698)

Updated docker/login-action to v3

* Reuse CriteriaQuery when retrieving result count. (#5702)

When migrating to the latest Hibernate we saw the following exception:
```
Caused by: java.lang.IllegalArgumentException: Already registered a copy: SqmBasicValuedSimplePath
```
This is because we were recreating a CriteriaQuery for our Queries.
Hibernate no longer allows us to do that, but rather allows us to use the existing
CriteriaQuery, but use the convenience method createCountQuery to
provide the criteria query for our createQuery.

Also removed 6.1.7 versionfor hibernate core  that allows us to use the Boot Bom.
This also caused some downstrem issues where dataflow was using the old version brought in from skipper

* Update spring-statemachine to 4.0.0 (#5707)

This should make skipper to start and run. 
Simple ticktock stream shouldwork ok.

* Update Code to use JobRepository bean directly

Since we will only support BOOT 3 we do not need the JobRepositoryContainer
to retrieve BOOT3 or BOOT 2 based JobRepositories.

Update test code to set the default time for the local date time sample to
a default of midnight

Allow services to use JobServie and JobExplorer directly

Currently we use containers to allocate the JobService and JobExplorer based on boot version.

This is no longer necessary.  So this PR removes these containers

Update Project as to restore tests success percentage to original state

After the updates below tests that were passing started to fail.
Some because of Batch 5 updates, but others because of the removal
of some of the container classes.

Update code based on code review comments

Removed todo
Removed ExtendsWith statement

* Adjust metrics replicator for Spring Boot 3

This commit adjusts the MetricsReplicationEnvironmentPostProcessor to account
for the Spring Boot 3 change in the metrics config props prefix scheme from
'management.metrics.export.<meter-registry>.<property-path>' to
'management.<meter-registry>.metrics.export.<property-path>'.

Update from PR review

* Update ci.yml free diskspace

* Update prometheus-rsocket-proxy to 1.6.0-SNAPSHOT

This commit updates the Prometheus RSocket proxy to 1.6.x which in turn is
updated to Spring Boot 3.2.x.

* Replace job execution and task batch containers with Task implementations

Replace JobExecutionDaoContainer with JdbcSearchableJobExecutionDao.
Replace TaskBatchDaoContainer with TaskBatchDao

Replace JdbcTaskExecutionDao with the Dao from task.

Add requested changes based on code review

* Use DataflowTaskExecutionMetadataDao in the place of DataflowTaskExecutionMetadataDaoContainer

Replace DataflowJobExecutionDaoContainer with dataflow's JobExecutionDao.

Replace DataflowTaskExecutionMetadataDaoContainer with DAO from dataflow

Replace TaskRepositoryContainer with TaskRepository

Move TaskRepository and SCDFDao beans from AggregateTaskConf to TaskConf

This is en-leu of the next phase where we begin to remove the aggreate configurations

Polish based on code review

* Adjust order of DefaultEnvironmentPostProcessor

This commit adjusts the change of the order attribute made in
commit de58247. The default EPP in fact needs
to run after the ConfigDataEnvironmentPostProcessor and before the
ConfigDataMissingEnvironmentPostProcessor.

* Fix skipper_manifest.data column oid to text conversion that was missed.

Fixes #5715

Cherry picked from 2.11.x

* Migrate Task Components to Boot 3, Task 3

This is Phase 1 of migrate the Task and Job components to Boot3/Task3/Batch5
Its purpose was to focus on migrating as much Task related features over as possible.

Status:
* All Task only tests in Spring Cloud Data Flow Server Core are passing
* Job related migrations have not started.  Nor those that are Batch/Task related
* Dataflow does start and a person can register a Task app and create a task definition.
* Task launches are still failing but Chris is looking to resolve that due a dependency hiccup from deployer

Noteable changes
* Establish property that tells which platformTransactionManager Task should use
* Update Task components to run Boot3 only components.
* Remove Schema related code from Task Launches and exploration
* Remove AggregateExecutionSupport from project and remove usages.
* AggregateTaskExplorer is still present and debating renaming it and keeping it because of some of the features it offered.
* Removed the AggregateTaskExecution

Updated based on code review

* Update Spring Cloud Deployer to 3.0.0-SNAPSHOT

* Use CLASSIC boot loader

This commit does the following:

* Configures the Spring Boot maven plugin to use the
legacy CLASSIC boot loader when launching/building uber jars

* Simplifies the BootClassLoaderFactory by removing support for
legacy Boot 1.3x jar format

* Remove extra Spring Cloud Task transaction manager

Prior to this commit there were 2 transaction managers defined.
One for Dataflow and one for Spring Cloud Task. The former was marked
@primary and it was causing confusion. This commit removes the task
specific one and uses the same manager for both cases. This is possible
by setting the 'spring.cloud.task.transaction-manager' property.

Move the Spring Cloud Task txn mgr property

This commit moves the SCT 'transaction-manager' property to the
dataflow-server-defaults.yml default properties file.

Add TransactionManagerCustomizationAutoConfiguration to DataFlowServerConfigurationTests contextRunner

* Upgrade spring-shell 3.2.2 (#5722)

This commit updates spring-shell to 3.2.2 and also does the following:

* Exclude spring-cloud-dataflow-common-persistence as it caused
  jdbc stuff in shell and then failures with datasource autoconfig
* Temporarily in tests use reflection as some methods in spring-shell
  are not public anymore.
* Add commented out shell log file settings which is a way to
  log hard shell startup errors

* Update Job to Batch 5 and remove schema usage

* Removed SchemaController/Tests/Documentation

* Removed the BatchVersion

* Service calls used by JobInstanceController have been migrated to batch 5.x

*  Migrated SQL Statements in AggregateDataFlowTaskExecutionQueryDao to use Task Execution

This is a first step to retire the use of the aggregates.
* DefaultTaskExecutionServiceTests - since we are no longer actively passing table prefixes the properties tests were dropped by 5
   * Also Removed BATCH_ and TASK_ tests
* JobExecutionControllerTests and JobExecutionThinControllerTests also had similar changes
* Some tests were either @disabled or the @test was commented out because the fixes need more investigation.
   * they were marked with todo
* We also need to revisit the use of locals over timezones.  The stop gap is to use the default locale.  This was marked with todo.

* Give SCDF ability to compile and run tests to completion

* Disable tests that need more investigation
* Refresh the TODO and Disable messages to reflect an accurate message of what is required
* These changes do not include those for skipper
* To compile to completion comment out spring-cloud-skipper module then execute mvn clean install

Update code based on review

* Update DAOs to no longer use aggregate

Update tests to remove Schema Version checks

* Update tests so that they no longer use aggregate views
* Update SimpleJobService MariaDB and PostgresDB to match was is in 2.11 tests
This was to resolve the sporadic connection pool problems

Remove AggregateJobQueryDao

* remove unused tests
* Update SimpleServiceFactory to remove use to the aggregate code

Rename AggregateTaskconfigruation/AggreageTaskExplorer to an accurate name

Rename aggregate package to composite

Remove uses of AppBootSchema and spring-cloud-dataflow-schema-core module

Re-enable more tests

Remove or update tests that are Boot3 specific

* Remove schema-core module from scdf-core module

* Added <argLine>--add-opens java.base/java.util=ALL-UNNAMED</argLine> to surefire-plugin and failsafe-plugin configurations. (#5753)

* Move DataflowTaskExplorer and associated classes to server core (#5754)

* Remove aggregate-task module
* Rename CompositeTaskExplorer to DataflowTaskExplorer

* Update versions for Skipper.
Change rest docs usage.

* Switch to non-blocking call in action

- UpgradeDeployTargetAppsSucceedAction can't use
  old non-reactive api's as we can't block within
  sm execution.

* Setup new apps for skipper tests

- Old manifests were using very old app version
  which don't run on jdk17 so add new
  definitions which uses latest app versions
  able to run on modern jdk's.

* Remove DataflowSqlPagingQueryUtils as it is unused

* SCDF does not need to support JSR-352 as of Batch 5.

Remove JobName search for JSR-352.

Address requested code review changees

* Resolve JobExecution Search by date test failure

We occassional fail to get the last entry of the test results
This is a verification if the test end time is too short

* Add MAX_START_WAIT_TIME property to CTR

Add documentation for CTR property MAX_START_WAIT_TIME.

Add description so it would appear as a properties option in the UI

Small polish

* stream update should not require --properties (#5785)

Currently a user can not set --propertiesFile option on a stream update command
This fix allows users to use --properties or --properties file with a stream
update command.
Added tests to verify

* Added TaskExecutionThinController and TaskExecutionThinResource.

Update TaskTemplate to changes in 2.11.x to improve link validation and reporting.

Remove andDo(print())

* Migrate SCDF-SINGLE-STEP-BATCH-JOB to Boot3 Batch 5

* Updated representation of start time and date given timezone.

* Temporarily disable Skpr auto config tests till full migration

* Version updates and cleanup on Skipper files.

Switch to jlumbroso/free-disk-space

remove docker

conditional docker login

find a way to get secrets in PR build

Skip tests on spring-cloud-skipper-core
Removed JMockit.

* Update classic doc to remove boot 3and boot schema

Remove the boot 3 and boot 2 schema changes
Also used queryParam when necessary per Boot3 requirements.

* A document test requires generateJumpToItemQuery that is not available

Disabling the test until the generateJumpToItemQuery is available

* DataflowSqlPagingQueryProvider (#5757)

* Implement DataflowSqlPagingQueryProvider by copying implementations from Spring Batch 4.x `org.springframework.batch.item.database.support`
Removed @disabled for tests that now function as expected.

* Remove docker login from ci-pr.yml

* Updated for requests

* Ensure docker compose will still work if docker-compose is not present.

* Update DB2 docker image uris.

* Updated DockerComposeTests and DockerTests.

* Upgrade spring-shell 3.2.4

* Provider for trying OCI accepts header when manifest result returns no config and schemaVersion less than 1.
Issue #5819

Test for DefaultContainerImageMetadataResolver needs to be updated to Jupiter and Assert4j

* Update to Spring Boot 3.3.0

* Update version of the apps metadata plugin used by the sample apps to
  get around issue of NoClassDefFound for spring-data-jdbc config props

* Modify MySQL57Database to use newly introduced `ensureSupported` API

* Replace use of `OAuth2ClientPropertiesRegistrationAdapter` with newly
  introduced `OAuth2ClientPropertiesMapper`

* Update to Flyway 10.10.0

The DB2 and Postgresql support now lives in its own module.

See https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.3-Release-Notes#flyway-10

* Update various dependencies

* Continue using prometheus 0.x client

See https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.3-Release-Notes#prometheus-client-1x

* the document plugin needs to use the current spring javadoc

When release SCDF be sure to update  javadoc-spring.version to the proper spring version

Updated based on code review request

* User needs ability to specify app version when creating schedule

This update allows user to specify version.label=<version.number>
Tests were also updated because the original settings assumed that the
 appregistry was real instance instead of being mocked. Thus the find would always return null.
And in this case the tests returned a false positive.
Now that the mocks are in place it excercises all the code.
Also added explicit test if user does not set the version number.  Some of the tests do this,
but wanted an explicit test to verify this.

resolves #5705

* [CI] Remove unused labeler workflow (#5840)

As part of the Spring Project assessment tool initiative we needed to
clean up our unused labels as they have grown out of control. We did
this manually in the Github UI. We will manage them in the UI going
forward.

(cherry picked from commit 2836b64)

* Add trailing slash to dashboard (#5811)

- Boot 3.x changed handling of trailing slash meaning
  it's now adviced to define both /dashboard and
  /dashboard/ in UiController

* Add JDK 21 and remove 8, 11 from build-images.
Remove JDK pre 17 from container tests.
Add JDK 21 to container tests.
Update DataflowOAuthIT to dump last error output from curl.

Update JDK list for containers in AbstractDataflowTests.

* Remove thinTaskExecutionList from TaskTemplate (#5842)

ThinTaskExecutions are not a part of SCDF-3.0.x.
During the migration some of the ThinTaskExecution URL calls were left in the rest client and this caused some test failures in IT.
This PR removes the API access from the client for ThinTaskExecutions.

* Migrate CTR to Boot 3.x and Batch 5.x (#5839)

* Migrate CTR to Boot 3.3 and Batch 5

* Remove the Batch Configurer and replace with a Configuration
* Update Tests so that they will work with Boot3
* Removed EnableBatchAutoConfiguration no longer needed if using BatchAutoConfiguration
* Removed schema requirements

* Re-enable CTR Module build in main pom.xml

* Add ability for composed task runner to use the proper JobRepository and TaskExecutor

* BeanPostProcessor has been added so that CTR can use its jobRepository vs. the one provided by BatchAutoConfiguration

* Update to prometheus-rsocket-proxy 2.0.0-M1 (#5888)

* Update Prometheus metrics prop names (Boot 3.x)

* Update Influx metrics prop names (Boot 3.x)

* Update metric prop names env vars (#5894)

The previous commits for updating metric names to the Boot 3.x format
missed some env vars. This commit updates the env vars as well.

* Update metric prop names in yaml files (#5895)

The previous commits for updating metric names to the Boot 3.x format
missed some entries in yaml files. This commit updates the yaml files.

* Update all module except spring-cloud-dataflow-common-test-docker to JUnit 5.
Converted JUnit Asserts to AssertJ
Remove all Hamcrest matchers except those required by mockMvc assertions.
Disabled failing tests for further investigation.

* Update gitignore for generated gradle files

Atleast vscode and possibly other ide's create paths
.github/workflows/download-jar/.gradle and
src/add-deps/.gradle which should not go to git.

* Add import order to vscode settings

* Remove jdk8 from spring-cloud-dataflow-server

May cause ide's to use wrong jdk as boot 3.x
requires jdk17.

* Need to select the serialization method for JobParameters for the commandline

User needs ability to set the default serialization technique for SCDF when restarting a job
User needs ability to select a serialization technique for a specific job restart
When user restarts a job repository from the list that is derived from thinjobexecutions it should use default technique

Add restful documentation.

Add support to allow user to set useJsonJobParameters for job relaunch via the shell.

Note: there are not tests for the shell update in this commit. This is because the current
set of tests rely on @EnableDataflowServer which does not work.   But before we fix
@EnableDataflowServer we need to make sure we want to carry it forward per Issue #1040

Polish PR before push to repo

JobCommand should ignore identifyingParameters when deserializing JobParameters

This is a generated list and and will cause deserialization to fail if not skipped

SCDF needs to support any type of class for JobParameters

However, if the class is not a base java type or one provided by dataflow the user has to build dataflow with their class included.

Add tests for JobParameterJacksonDeserializer

Remove Disabled annotation from JobExecutionController tests that are no longer needed

Update per code review request.

Added test for JobParamterMixin via the JobTemplateTests
Removed the duration parameter from getDurationBasedEstimate method
Rebased

Reset the duration calculation from nanos back to millis.

Optimized restartExecutionArgs routine that removes duplicates.

This was per a comment in code review

Remove unnecessary exclusions from AbstractShellIntegrationTest

The changes are  code review requests.

Add warn log message when job restart id is invalid.

* Initialize ObjectMapper with the tools provided by DataFlowTemplate when testing

* These changes were identified during code review

* Fix rest docs tests. Updated to usage of latest stream apps and tasks.

* Remove .jdk8 files. (#5900)

* Fix more previously disabled tests.

* Update SCDF so that multipleComposedTaskWithArguments succeeds

Step 1. Make sure to remove the version from the docker compose.  It is no longer needed and causes older versions of docker to fail
Step 2. Update compose files to use the latest version of SCDF 3.x instead of 2.11.x
Step 3. Update build image script so that uses java 17 when creating containers

Update the DataFlowIT and the Abstract classes it is built on so that multipleComposedTaskWithArguments test passes.

Notice that JobParameterJacksonDeserializer and JobParametersJacksonMixIn have been updated.  These changes mirror those in #5850.   These were required for the test to pass.  At the time this PR is merged we can merge accepting those from #5850.

Provide docs on how SCDF images are created and pushed

Also update the DEFAULT_JDK to Java 17

Update PR based on code review comments

* Added log message in case a JobParameter Type is invalid
* cleaned up workflow.adoc

* Remove .jdk8 files. (#5900)

* Update SCDF so that multipleComposedTaskWithArguments succeeds

Step 1. Make sure to remove the version from the docker compose.  It is no longer needed and causes older versions of docker to fail
Step 2. Update compose files to use the latest version of SCDF 3.x instead of 2.11.x
Step 3. Update build image script so that uses java 17 when creating containers

Update the DataFlowIT and the Abstract classes it is built on so that multipleComposedTaskWithArguments test passes.

Notice that JobParameterJacksonDeserializer and JobParametersJacksonMixIn have been updated.  These changes mirror those in #5850.   These were required for the test to pass.  At the time this PR is merged we can merge accepting those from #5850.

Provide docs on how SCDF images are created and pushed

Also update the DEFAULT_JDK to Java 17

Update PR based on code review comments

* Added log message in case a JobParameter Type is invalid
* cleaned up workflow.adoc

* Update apps version refs to 5.0.0 / 3.0.0 (#5896)

* Update stream apps version refs to 5.0.0
* Update task apps version refs to 3.0.0

See #5897

* Update with merge from main-3

* Remove metrics auto-config exclusions (#5902)

This commit removes temporary auto-config exclusions that were in place
to allow compiling against Boot 3.x prior to migrating to the newer
Observation API.

* Update documentation for initContainers addition in spring-cloud/spring-cloud-deployer#465 [skip ci]

Update paragraph for multiple Init Containers.

Fix mariadb mount to be /var/lib/mysql (#5875)

Fixes #5877

Add spaces after columns for clarity.  This is done during merge

* SCDF should be able to migrate a schema from 2.10 to 3.0

Currently the migration fails with a validation error.
The cause of the validation error was that some commits from 2.11.x that contained flyway migration scripts were not ported to the main-3 branch.

The following commits that contained flyway migrations were migrated to main-3 from the main branch.
* 62ea6c5
* 6f97589

The goal of this PR is to resolve the validation error so DB migrations will work properly.

A subsequent PR will be submitted will add the feature code for commit 6f97589.

* Forward port ctr status on Thin Controller. (#5906)

* Added ThinTaskExecution to store ctrStatus.
Update Controller and Service to populate the ctr status.

Fixes #5907

* Updated for comments.

* Enable MariadbSharedDbIT (#5908)

This commit re-enables the `MariadbSharedDbIT` and disables the
Java 21 test variant as Java 21 container images are not yet supported.

* Update tracing props (#5925)

* Update Prometheus Rsocket property name

This property rename was missed during the initial update to Prometheus
Rsocket 2.0.0-M1 (b61c3c9).

* Update tracing props in Docker Compose files

This updates the Wavefront and Zipkin docker compose templates
with the properly named tracing related properties to enable traces exported
for each.

* Fix substitution in github-release.yml [skip ci]

* Fix tracing properties in docker compose (#5946)

This commit adds the global tracing property
'management.tracing.enabled' to the Zipkin and Wavefront docker compose
files.

Also, the Wavefront docker compose file excludes the Zipkin tracing
auto-configuration and the Zipkin docker compose file excludes the
Wavefront tracing auto-configuration.

* Updates for JUnit5 and AssertJ

* CI requires Java 17.

* Updated based on comments.

* Updated based on comments.
Added TODO on comments.

* Fixed imports

---------

Co-authored-by: Chris Bono <cbono@vmware.com>
Co-authored-by: Glenn Renfro <grenfro@vmware.com>
Co-authored-by: Chris Bono <chris.bono@gmail.com>
Co-authored-by: Janne Valkealahti <janne.valkealahti@gmail.com>
Co-authored-by: Clemens L <clemens-l@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants