Questions tagged [spring-cloud-dataflow]

Spring Cloud Data Flow is a toolkit for building data integration and real-time data processing pipelines. Pipelines consist of Spring Boot apps, built using the Spring Cloud Stream or Spring Cloud Task microservice frameworks. This makes Spring Cloud Data Flow suitable for a range of data processing use cases, from import/export to event streaming and predictive analytics.

Filter by
Sorted by
Tagged with
0 votes
1 answer
10 views

Spring cloud dataflow's dashboard not found - 404

i'm trying to create a spring cloud dataflow server v2.10.3, but when i run the project i can't access the dashboard, it gives me an 404 error... here is my main class : @EnableDataFlowServer @...
Fahd's user avatar
  • 1
0 votes
0 answers
23 views

Spring Cloud Dataflow - Composite Task Runner Issue

I'm exploring all the features of Spring Cloud Dataflow for our project. Currently I'm doing all development locally(No docker, Cloudfoundry & Kubernates) Currently I have created 2 Spring Batch ...
iCode's user avatar
  • 8,892
0 votes
1 answer
35 views

Stream deployment properties are being locked after a deployment with errors

We're using spring cloud dataflow 2.10.3 deployed using helm in a kubernetes cluster. I'm not sure if this is a bug or something were doing wrong but we've seen it happen across all types of streams, ...
oldvirus_231's user avatar
0 votes
1 answer
22 views

Redis serializer properties in SCDF

I'm using a splitter and an aggregator in SCDF 2.10.3 and I want to cache messages in Redis. Here is a simple example stream : http | splitter | aggregator | log This stream works fine without Redis. ...
CEDDM's user avatar
  • 21
0 votes
0 answers
18 views

SCDF server returning "Error waiting to get Release from a statemachine" when trying to deploy scdf stream

I am trying to deploy an scdf stream through ArgoCD but the pod is not created and I receive the following error: Error waiting to get Release from a statemachine Any idea what this error is? Where ...
goulpan's user avatar
  • 466
0 votes
1 answer
29 views

Spring batch task pod(jvm) is not automatically terminated after competition of task

Spring batch task pod(jvm) is not automatically terminated after the competition of the task. So, I thought to close the application context explicitly post-completion of the task by closing the ...
Ankit Gupta's user avatar
0 votes
1 answer
34 views

Spring Cloud Data Flow 2.10.3 : may i launch a Task linked to a specific application version, instead of default version

I'd like to run a Task (or create a new Task and run it) whom execution will trigger deployment and execution of a specified (on task level) version of the linked application. I don't find how to do, ...
Damien Tacheron's user avatar
0 votes
0 answers
29 views

Spring Cloud Data Flow deploy batch application to multiple cluster

I have two clusters in Openshift that share the same database: Cluster A: https://10.0.0.1:443 Cluster B: https://10.0.0.2:443 The SCDF is installed on cluster A where I can deploy batch ...
Kane's user avatar
  • 9
0 votes
0 answers
16 views

Spring cloud task(composed Task) restart from the failed job

I have a spring cloud composed task which has nearly 10 jobs . If any of the job fails from the composed task, I have to continue the flow from the failed job. For Eg: After launching the composed ...
17_4088_ Subashini.S's user avatar
0 votes
0 answers
33 views

SCDF Task Kubernetes Persistent Volume attachment

I have created a simple spring cloud task project, which will generate a file and add to local folder. I am using SCDF(Spring cloud data flow) and kubernetes for deployment. I need to generate file to ...
ThunderBoy's user avatar
0 votes
1 answer
26 views

Spring cloud stream: exception in reactive pipeline crashes stream with error Dispatcher has no subscribers

I have below consumer function. It works if the WebClient call is successful @Bean fun consumer() = Consumer<Flux<String>> { flux -> flux.concatMap { s -> WebClient ...
user3908406's user avatar
  • 1,433
0 votes
0 answers
7 views

Sync application Stream Source in Spring Cloud Data Flow

I am designing the Business to Business data stream pipeline with Spring Cloud Dataflow Stream application. HTTP source to Http sync stream pipeline with async. How we can create the design with sync ...
Zishnu's user avatar
  • 1
-1 votes
1 answer
26 views

Connection to rabbitmq failed when deploying spring cloud data flow in the local

I referenced the documentation: https://dataflow.spring.io/docs/installation/local/manual/ to run the spring cloud data flow in my local mac, and then deployed an http | log stream according to https:/...
Jeff Tian's user avatar
  • 5,246
0 votes
0 answers
7 views

DataflowTemplate makes HTTP Call at constructor call

The DataFlowTemplate that I am using in order to start tasks deployed to Spring Cloud Dataflow executes an HTTP Call to the SCDF server when I am calling the constructor, e.g. new DataFlowTemplate(...)...
Manu's user avatar
  • 284
0 votes
0 answers
19 views

SCDF on k8s: Configure Dataflow Server to start Task Container with volume

I'm trying to find a way to configure the Spring Cloud Dataflow Server, which I'm deploying to kubernetes, so that it mounts a volume into the pods it creates whenever I am starting a task. I got an ...
Manu's user avatar
  • 284
0 votes
0 answers
24 views

Spring cloud stream and async process

I have below simple consumer app. I wonder how to make it so that consumer pull the next message only after the current async process is completed @SpringBootApplication public class Application { ...
user3908406's user avatar
  • 1,433
1 vote
0 answers
100 views

Spring Cloud Data Flow (version 2.11.0) is failing at launch with DB2 instance (DB2 v11.5)

After building the SCDF with custom Db2 driver, while trying to launch the server (spring-cloud-dataflow-server-2.11.0-SNAPSHOT) is failing with: Caused by: org.springframework.cloud.dataflow.common....
Subhayu's user avatar
  • 21
0 votes
0 answers
12 views

Which has better performance Message<String> or Message<byte[]>

I am using spring Cloud stream Functional Model If someone can help understanding which has better performance Function<Message, Message> or Function<Message, Message> OR Function<...
user1421204's user avatar
0 votes
0 answers
22 views

SCDF with Spring Batch 5 on k8s: Task Container relies on Resource created by previous Task

Background I am using Spring Cloud Dataflow to execute a Task which basically is a Spring Batch Application. This Batch Application consists of multiple jobs - one of them creates a file that can then ...
Manu's user avatar
  • 284
0 votes
0 answers
14 views

Spring Cloud Dataflow on Kubernetes - Startupprobe property

Am trying to setup SCDF on kubernetes using the helm chart https://github.com/bitnami/charts/blob/main/bitnami/spring-cloud-dataflow/README.md Am able to get this working and deploy streams. Question ...
CuriousK's user avatar
0 votes
0 answers
29 views

Spring Data Flow - Maven setup using Docker Compose

I have installed Spring data flow using docker compose, when i try to build/run something i got an error referring to (https://repo.spring.io/libs-snapshot): status code: 401,. I know that the link ...
Frenk's user avatar
  • 71
0 votes
1 answer
19 views

Can't build scdf from source, missing dependency : io.pivotal:pivotal-cloudfoundry-client-reactor:jar:2.1.0.RELEASE

In maven central repo, there is only 2.2.0-Relase which is available. https://central.sonatype.com/artifact/io.pivotal/pivotal-cloudfoundry-client-reactor/2.2.0.RELEASE/versions But scdf has a ...
Damien Tacheron's user avatar
0 votes
1 answer
86 views

Versions problem with custom app and scdf

I'm new to scdf, I've set up a local server, with the help of a guide text my docker-compose custom env with which it rises services: dataflow-server: environment: - maven.remote-...
MyName's user avatar
  • 1
0 votes
0 answers
14 views

VertexAI online prediction with timeseries data sliding windows

I already have a trained model (that does classification) that takes as input timeseries of shape [2500, 3]. I exported it on vertex as a custom model and created a public endpoint from it. I can ...
Sebastien Benoit's user avatar
0 votes
2 answers
42 views

Bitnami Springcloud dataflow deployment error in kubernetes using external mysql database

I am deploying spring cloud dataflow stream from artifacthub.io using bitnami chart. I am facing issye with JDBC driver and flyway error during pod startup in the latest chart but is working fine in ...
Anish Raut's user avatar
0 votes
0 answers
12 views

pring-cloud-skipper-server upgradation issue

I am using a stack under that having an spring-cloud-skipper-server container previous version was 2.6.1 tried to upgrade to 2.9.2 from that container was not in up state having this error message Err&...
saikiran's user avatar
0 votes
1 answer
41 views

Spring Cloud Dataflow Stream - org.springframework.cloud.skipper.SkipperException: Can not find a package named

While deploying Streams in SCDF 2.9.2 with skipper 2.8.2, I observed frequent scenarios, wherein if a stream deployment fails for some reason, I cannot destroy the stream definition or re-deploy the ...
Aravind1986's user avatar
0 votes
1 answer
63 views

MariaDB - supply DER formatted CA with serverSslCert

I am building a MariaDB connection string where according to the following documentation I would like to supply the CA certificate directly in the string rather than a file: https://mariadb.com/kb/en/...
adesai's user avatar
  • 390
0 votes
1 answer
53 views

Spring Data Flow - Log Sink - How to log plain text message

I have been following the spring tutorial to create a simple http ingest stream: https://dataflow.spring.io/docs/stream-developer-guides/getting-started/stream/ I define and deploy a stream: http --...
Oliver Reimer's user avatar
0 votes
0 answers
48 views

Reactive programming and SCDF Kafka stream on high load

We are using a SCDF Stream. It is consuming from Kafka topic via a bean that is a providing consumer. After reading topic events, we are trying to process those events like this. KafkaUtils....
Barış Vural's user avatar
0 votes
1 answer
106 views

Spring Cloud Dataflow - Docker compose in windows local - Failed to resolve MavenResource

Running spring cloud dataflow docker compose in my windows local. Created http|log stream using below command stream create --definition "http --server.port=8080 | log" --name myhttpstream ...
Prabakaran p's user avatar
0 votes
1 answer
24 views

Migrating Spring cloud data flow server with Oracle Database

I am trying to migrate spring-cloud-dataflow server that is currently using h2 database with oracle database. I modified the spring-cloud-dataflow.jar by adding ojdbc jar into boot-inf/lib directory ...
Hritik Sharma's user avatar
0 votes
1 answer
57 views

How to integrate Spring Cloud Data Flow with Spring Boot Admin for batch applications monitoring?

Can we integrate Spring Cloud Data Flow with Spring Boot Admin. I am new to these technologies so some additional enlightenment will be appreciated. We have a Spring Boot Admin in place to monitor ...
Kranti Man Singh's user avatar
1 vote
1 answer
142 views

Migrating Spring Cloud Connectors to java-cfenv in Cloudfoundry causes TLS failure with Rabbit-MQ service

While upgrading Spring Cloud Services from 2.4.1 to 3.5.5, we are seeing issues with the app not able to connect to Rabbit MQ service in cloud foundry. It throws the following exception:- c.r.client....
Aravind1986's user avatar
0 votes
0 answers
39 views

How to customize bindings/queues/Exchange behaviours correctly on Spring Cloud Dataflow custom app

I'm developing a custom app who inject the main function of time OOTB app, but I want to customize also the behaviour of the queues and Exchanges created at the moment of deploying a stream on SCDF, ...
Emmanuel_Arreola's user avatar
0 votes
1 answer
57 views

Spring Cloud Dataflow custom stream application metrics aren't showing up in Grafana

SCDF version 2.10.2 I can't get my custom source, processor or sink to expose metrics to my prometheus/grafana setup. I've followed the custom stream application tutorial here: https://dataflow....
oldvirus_231's user avatar
0 votes
1 answer
39 views

in SCDF(Spring Cloud Data Flow), how can i deploy application(stream or task) as statefulset? not deployment

in SCDF(Spring Cloud Data Flow), how can i deploy application(stream or task) as statefulset? not deployment I could find some properties related to statefulset (ex: statefulsetInitcontainer, ...
Kim-menu's user avatar
0 votes
1 answer
50 views

No properties shown in Data Flow Server for custom stream application

I am running Spring Data Flow Server 2.11.0-SNAPSHOT (the same problem occurs for 2.9.1, too) within a docker compose environment. I have an issue with custom stream app properties, which are not ...
Ralf Kunkel's user avatar
0 votes
1 answer
24 views

Spring cloud data flow one processor sending customs data to mutiples sinks

In Spring SCDF, I got one processor with just one input and I want to send data to 3 different sinks depending on data type for example: integers to sink1, Strings to Sink3 and Longs to SInk3 , each ...
Georges Tieno's user avatar
0 votes
1 answer
149 views

Spring cloud streamBridge to Kafka delivery check

I have a few requirements for the spring cloud stream I have: it needs to take a KStream from a single Kafka topic on one cluster and send a message to multiple topics on another cluster. On some ...
Duella's user avatar
  • 1
0 votes
0 answers
28 views

SCDF launching spring batch Pod in kubernetes : Pod resource utilization

Apologies if this is a duplicate question - I looked out for similar ones but did not find exactly what I was looking for. Here is the situation : I have to upload file and run spring batch to process ...
saiD's user avatar
  • 19
0 votes
0 answers
30 views

Accessibility standards with Spring Cloud Dataflow UI

Do we know if the Spring Cloud Dataflow UI supports any accessibility standards? Ideally WCAG 2.1 standards.
adesai's user avatar
  • 390
0 votes
1 answer
38 views

SCDF REST API. List of Task Schedules is empty

According to SCDF documentation, the /tasks/schedules endpoint should return a list of Task Schedules. However, the returned list is empty despite the present schedules, which are visible UI https://...
Viktor's user avatar
  • 345
0 votes
0 answers
267 views

onConfiguration$FunctionBindingRegistrar : You have defined function definition that does not exist: mongoSink

basically i am switching mongo sink(consumer) app from annotation based approach to functional model using consumer bean. New code @Bean(name="input") public Consumer<Message<?>>...
shiva krishna's user avatar
0 votes
1 answer
50 views

How to identify Current Task Execution Count in Spring cloud dataflow?

I have explore the official documentation https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-current-count-request-structure Here no ...
mukul verma's user avatar
0 votes
0 answers
47 views

spring-cloud-dataflow OAUTH2 roles and permissions

I am trying to setup Oauth with scdf 2.9.2 and I am at the point where I am able to get the ID-token with the authorization_code grant type. But my user has no roles, in the token our roles are under ...
jakeops's user avatar
0 votes
1 answer
54 views

Spring Cloud Dataflow deployment name change every time when stream un-deploy and deploy again in Kubernetes

We have deployed the pod of Spring Cloud dataflow and Skipper in AWS EKS. We have registered the application and created the stream. Whenever we are undeploy and deploy the stream that time deployment ...
Anish Raut's user avatar
0 votes
0 answers
63 views

How to handle multiple request with Spring cloud data flow deployed in K8?

We are trying to develop production ready application using spring batch and spring cloud data flow server. We came across a issue when users will submit multiple files to process to scdf server at a ...
mukul verma's user avatar
1 vote
0 answers
109 views

Spring cloud stream not recover channel on timed out error

I know it uses spring-amqp to handle recovery. One case I found where it doesn't recover the channel is that if a spring cloud stream app couldn't process message within 30 mins (default timeout for a ...
user3908406's user avatar
  • 1,433
0 votes
1 answer
22 views

Is there any way to enable Email alertes in scdf

I am using latest version of speing cloud data flow. Like to enable email alerts when an event occur in SCDF. Is there any way we can customize or configure it with inbuit functionality in scdf or ...
user avatar

1
2 3 4 5
26