Streamlined Deployment of ETL Workloads in Spring Cloud Data Flow via Infrastructure as a Code (IaC) #6114
Description
Problem description:
We are starting to experiment with Spring Cloud Data Flow and would like to know the recommended way to deploy workloads (e.g., tasks, batches, and streams) from the CI/CD pipeline. The CLI commands may not be very portable or convenient for us, and frequent changes to the RESTful APIs could require significant effort to upgrade and fix API calls with each new version.
Solution description:
Ideally, we would prefer an SDK to manage deployments. This solution would allow us to define the desired final state of our ETL workloads (composed of tasks, batches, streams, etc.) as code, following the infrastructure as code (IaC) approach, and deploy them via the CI/CD pipeline. This would help us avoid issues related to CLI and RESTful API changes.
Description of alternatives:
We have considered using the CLI commands and RESTful APIs, but these options may not be as portable or convenient, and they could require frequent updates to keep up with changes in new versions.
Additional context:
We are looking for a solution that minimizes the need for manual updates and provides a streamlined process for deploying ETL workloads in a CI/CD pipeline environment.