Questions tagged [spring-xd]
Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. The project’s goal is to simplify the development of big data applications.
565
questions
0
votes
1
answer
105
views
Can't install Spring-XD properly on Windows
For using batch processing and some analysis, I'm trying to start some Spring-XD projects. I have worked with Spring-Boot previously. Sorry for my ignorance. I have followed official guide.
As a ...
0
votes
1
answer
757
views
Not able to convert Message from Json String to Object. ClassCastException
I'm using SpringXD where I have a Rabbit Source with outputType application/json. Next module receive it and convert it to Java Object. RabbitMq is my transport bus.
My configuration is pretty ...
0
votes
0
answers
66
views
SpringXD Kryo-shaded-3.0.3.jar and NoClassDefFoundError and ClassNotFoundException And Enum
I just noticed that we are facing Kryo Issues with a new payload in one of our new Streams in Spring XD (We are planning to jump to Spring Cloud Data Flow soon, so suggestion based on it, might no be ...
0
votes
2
answers
628
views
kafka consumer get a message which was consumed before
I have a consumer job in XD which will be completed once it receives a message which is produced by other producer job. And I trigger these jobs every day.
I found sometimes this consumer got a ...
0
votes
1
answer
26
views
Highest RabbitMq compatibility as Message Bus for Spring XD 1.3.0.RELEASE
I'm trying to figure out what the highest version of RabbitMq that I can use with Spring XD 1.3.0.RELEASE.
Basically we are using Rabbit as a message bus in Spring XD. Also we are using it as a ...
0
votes
0
answers
22
views
In a Spring-XD job module, how to write data to two files respectively at the same path in two servers where XD containers are running?
My XD cluster has two node servers, so there're two XDadmins and two XDcontainers. When I launch the job in XD-shell through one Admin server, the job writes data to the file at one node. At the other ...
3
votes
1
answer
6k
views
An illegal reflective access operation has occurred while setting up spring xd
i was following the guide Spring XD Guide to setup spring xd on window 7. When i tried to start the runtime and the XD shell and entered ./xd-singlenode, it gave me the following warnings and errors:
...
2
votes
1
answer
1k
views
reading CSV file from s3 using spark
I am new to Spark. I have a scenario where I need to read and process CSV file from AWS s3. This file is generated on daily basis, so I need to read and process it and dump data into Postgres.
I ...
0
votes
1
answer
46
views
FTP source - obtaining file reference each time a file is transferred
lets say i create a stream like this:
stream create --name ftpstream --definition "ftp --username=ftpuser --password=pass123 --host=host.com --remoteDir=/home/ftpuser --mode=ref | log" --deploy
If ...
4
votes
0
answers
3k
views
Class org.apache.kafka.common.serialization.StringSerializer not found
I wrote a Kafka producer that runs well when I execute it in anyplace besides Spring-XD daemon.
private Properties setProperties () {
Properties props = new Properties();
props.put(...
1
vote
0
answers
72
views
I am trying to run a Groovy script from Spring XD shell. But script does not trigger
I am trying to extract data from SQL SERVER into HDFS using groovy script.
The stream definition deploys the stream bu the script is not getting fetched after deployment.
Stream definition :
stream ...
0
votes
1
answer
32
views
springxd JMS sink acknowledge propery
I am unable to find any documentation on how I can use/customize the springxd JMS source acknowledge property.
Documentation only says -
JMS with Options
The jms source has the following options:
...
1
vote
0
answers
82
views
How to make the job restartable in Spring xD
The jobs under Executions section of Jobs in Spring xd admin ui has disable restart button. How will this get enabled? I tried making the job itself restartable by supplying the restartable="true"
1
vote
2
answers
72
views
Spring-xd Redis sink with Collection type MAP
Im trying to implement Redis sink with specific to collection type = MAP
stream create tTEST_GF_SINK --definition "trigger --initialDelay=0 --fixedDelay=1 --timeUnit=MINUTES --payload= 'new Date()....
1
vote
1
answer
396
views
Headers in http source Spring cloud flow
I am using postman to POST a http request with a file in form-data and want to pass the filename as a header. With file as a sink i want to get that file and save it in the name of the header value ...
1
vote
1
answer
38
views
springxd stream using HDFS-Dataset to save avro data unable to renew kerberos ticket
I have created a springxd stream ====> source-JMS queue -> Transform-Custom Java Processor (XML to AVRO) -> Sink - HDFS-Dataset.
Stream works perfectly fine but after 24 hours, since its continuous ...
0
votes
2
answers
609
views
Gemfire CacheListenerAdapter implementation ClassNotFoundException: org.apache.geode.pdx.internal.PdxType
I'm getting the following error when I tried to implement CacheListenerAdapter from gemfire.
java.lang.ClassNotFoundException: org.apache.geode.pdx.internal.PdxType
I am using the following code for ...
0
votes
1
answer
38
views
How we can add a field to JSON stream in spring XD?
How I can add a timestamp field if it doesn't exist in JSON stream in spring xd, I think we can use a using groovy script for that.
Suppose my JSON stream is {node : value} I want to add one more ...
0
votes
1
answer
109
views
Method call: Method toObject(java.util.LinkedHashMap) cannot be found on org.springframework.integration.x.gemfire.JsonStringToObjectTransformer
I am facing the following issue in Spring XD and gemfire
nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1004E:(pos 8): Method call: Method toObject(java.util....
0
votes
2
answers
180
views
Spring XD unable to find class definition for RabbitMQ ConnectionFactory
I am in the process of switching from xd-singlenode mode to distributed, using version 1.3.1-RELEASE.
I am running Spring XD in a docker container, with a Dockerfile similar to the one here. ...
0
votes
1
answer
111
views
spring-integration-kafka 1.3.1.RELEASE : Any workaround to use org.apache.kafka.common.serialization?
I'm using Spring XD to play with this issue:
Is there any workaround to deserialize messages that I'm receiving from a Kafka topic (0.9 version).
I have been trying to modify the dependencies to ...
0
votes
1
answer
162
views
Migration from SpringXD to Spring cloud data flow
we are currently using SpringXD to ingest the data from the various data sources to the sinks. Since Springxd is scheduled for end of availability, we would like to migrate to Spring Cloud data flow.
...
0
votes
1
answer
992
views
Spring Data Flow w/ 2 sources feeding one processor/sink
I'm looking for some advice on setting up a Spring Data Flow stream for a specific use case.
My use case:
I have 2 RDBMS and I need to compare the results of queries run against each. The queries ...
1
vote
1
answer
856
views
Spring Integration & Kafka Consumer: Stop message-driven-channel-adapter right after records are sucessfully fetched
I'm using the following confing:
spring-integration-kafka 2.1.0.RELEASE
kafka-clients 0.10.0.1
Kafka 0.10.x.x
spring-xd-1.3.1.RELEASE
I created my custom Kafka-source module for SpringXD. I set my ...
0
votes
1
answer
2k
views
SpringXD: auto-startup=false not working on Kafka message-driven-channel-adapter
I'm using SpringXD, and I have the following config:
spring-integration-kafka 2.1.0.RELEASE
kafka-clients 0.10.0.1
Kafka 0.10.x.x
spring-xd-1.3.1.RELEASE
I have the following config in my xml file:
...
0
votes
2
answers
162
views
SpringXD Error when deploying Stream: StringDeserializer class could not be found
I'm configuring my custom kafka source and I'm getting an error related to the value.deserializer kafka property.
This is the config that I have:
<!--Consumer -->
<bean id="container1"
...
1
vote
1
answer
58
views
SpringXD/Spring Integration: Using different versions of spring-integration-kafka for producer and consumer
I have the following configuration:
Spring-integration-kafka 1.3.1.RELEASE
I have a custom kafka-sink and a custom kafka-source
The configuration I want to have:
I'd like to still using Spring-...
0
votes
1
answer
175
views
Spring XD job deployment error
I have observed this error multiple times, when XD shell / XD-container is unexpectedly quit and the job is still in deployed state. When xd container, and xd-shell are restarted, its not allowing us ...
0
votes
1
answer
287
views
Spring XD integration test fails with "Path must start with / character"
I've been looking for a way to properly test my custom modules within Spring XD, so I've been trying to mimic how Spring Xd itself tests spark processors. However, no matter how I configure with ...
2
votes
2
answers
1k
views
How to route messages from kafka-sink to multiple topics
I have a spring-xd http-processor module with http-outbound-gateway that has a errorChannel and outputChannel. Any message with HTTP 200 comes to outputChannel, and rest of them land in failureChannel....
1
vote
1
answer
16
views
SpringXD - It is possible to scan different dirs via File source?
I've these dirs
|- Dad Dir
|--- Son Dir 1
|---- file1.txt
|---- file2.txt
|--- Son Dir 2
|---- file3.txt
it is possibile with a single stream, using ...
1
vote
1
answer
294
views
Spring XD - Logical operators in SpEL expression and filters
I would to create a filter like this
file --outputType=text/plain --dir=someDir --mode=lines | filter --expression='payload.contains(\"request url\") AND payload.contains(\"request ...
0
votes
1
answer
199
views
SpringXD and Spring Integration: Read from kafka topic every X minutes, then send to another Topic
I'm trying to implement a solution to create a SpringXD stream composed by a kafka source, a bridge module, and a kafka sink.
So I have something like:
<channel id="pollable">
<queue /&...
0
votes
1
answer
125
views
How can we send Spring-xd shell commands from Batch Job
Im trying to relaunch a JobB by running JobA. I was trying to use jobOperater.restart() and I was unsuccessful. I was able to restart the jobB from the xd-shell by "job execution restart --id 0". Can ...
0
votes
1
answer
59
views
SpringXD and Integration: Bridge, polleable &subscribable channels
I'm reading the Spring Integration documentation to figure out how to implment a bridge with polleable and subscribable channels (which will be one of my SpringXD modules).
So the documentation ...
0
votes
1
answer
127
views
On Spring-XD analytic-pmml stream, publishing json data via mqtt broker error?
this is my stream definition-
xd:>stream create test --definition "mqtt --url='tcp://localhost:1883' --topics='irisPayload'| analytic-pmml --location='/home/andy/Desktop/iris-flower-naive-bayes....
0
votes
1
answer
91
views
stream creation with analytic module in spring-xd get failed
I am working on spring-xd framework. I am following Spring-XD guide Analytic tab and want to deploy my prediction model for live streaming data.
I have built the iris classification model using ...
0
votes
1
answer
131
views
how to register springxd source mqtt to mqtt broker
I had created streams with spring-Xd such as-
stream create mqtttestfile --definition "mqtt --url='tcp://localhost:1883' --topics='helloTopic' | file" --deploy
Created and deployed new stream '...
0
votes
0
answers
104
views
SpringXD Job is still executing even after forceful Server shutdown
I'm trying to run a Job on SpringXD and the job has 7 steps, When I reach step 4 i'm shutting down the XD server. When I look into the Metadata of SpringXD I'm saving on OracleDB. I see that the Step ...
0
votes
1
answer
65
views
Spring XD: stream and source are inbound/outbound channel adapters?
The source component in Spring XD is designed always like a inbound channel adapter by Spring Integration?
Same question about sink component: is always designes like outbound channel adapter?
In ...
0
votes
1
answer
949
views
Getting springframework FatalBeanException
Hi I am implemented Spring-xd framework and I have added xml configuration in my spring-module.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/...
1
vote
1
answer
156
views
Spring XD - Could not resolve placeholder 'xd.module.sequence'
I am dealing with an issue similar to the following posts:
Spring -XD Stream deployment failure
issue in spring xd cluster when deploying my module
Also, saw that a JIRA ticket was opened:
https://...
0
votes
1
answer
74
views
Spring XD 1.3.1 stream with Cassandra 3.0
I am using DSE 5.0.5 which come with Cassandra 3.0.11
I am trying to use Spring XD 1.3.1 to connect to the Cassandra
I have a processor module which processes the data and a sink which actually ...
1
vote
1
answer
443
views
How to add new header from spring-module.xml
I need to add header from spring-module.xml. I am getting below response
headers:
file_name: sample.txt
content_type: text/plain
Payload: { "json":{ "type":"abcd","value":"1234" }}
In the ...
1
vote
1
answer
4k
views
java.lang.NoClassDefFoundError: com/sap/conn/jco/JCoException Error
java.lang.NoClassDefFoundError: com/sap/conn/jco/JCoException
at testSAP.TestSAP.execute(TestSAP.java:41) ~[na:na]
at org.springframework.batch.core.step.tasklet.TaskletStep$...
0
votes
1
answer
69
views
How to develop a spring XD custom job to download a file from SFTP location and save it on local machine
I'm trying to develop a custom job to download a file from SFTP location using spring XD. It'll really help if you could give me some guidance on this as I'm pretty much new on this.
0
votes
0
answers
411
views
Spring XD: Problems running Batch Job. "Error creating bean with name 'org.springframework.batch.integration.launch.JobLaunchingMessageHandler"
I'm trying to run my first job in spring XD. I deployed the job, and I created a stream which includes a cron, which will trigger my Job. This is what I have so far:
this is my xml config:
<bean ...
0
votes
1
answer
825
views
Spring Step Scheduling
I'm trying to schedule steps in a Spring Batch job. I tried using . But When I'm trying to deploy the job on SpringXd its failing. Below is my code and error I'm facing
<batch:job id="addB" ...
1
vote
0
answers
108
views
Using custom SpEL method in spring XD stream definition
My idea is to create custom SpEL method and register that in a spring XD module.
I have following in my spring XD module XML
<int:spel-function class="com.digitalmessagingsystem.batch....
0
votes
1
answer
149
views
Exhausted messages in Spring XD with RabbitMQ as the message bus
I have a Spring XD 1.2.0 installation with one server and a couple of containers using RabbitMQ as the message bus and I see lots of WARN messages in the logs such as these ones:
2017-02-05T22:51:03+...