8,599
questions
0
votes
0
answers
22
views
Spring-batch - read and write in the same table in batches (pagination)
How can I use (or is there a way) to use spring-batch to read and write in the same table in batches?
For context: I have a table that has millions of registers. From time to time many of those ...
0
votes
1
answer
28
views
org.springframework.batch.item.WriterNotOpenException: Writer must be open before it can be written to - in Spring batch Multithreading?
I've used FlatFileItemReader with the Classifier and ClassifierCompositeItemWriter in my project where we read Tab delimited file and convert it into CSV as a output file by performing some validation,...
-1
votes
1
answer
41
views
Database not initialized when executing end-to-end tests
I'm trying to build an end-to-end test on a spring-batch project, however, when trying to run the execution of a single step, the database appears to be not initialized. The contextLoads test runs ...
0
votes
1
answer
25
views
spring batch job using less than 1 cpu despite thread pool of size 20
I have below step configured that is running super slow. On our own servers it runs in less than 20 minutes but in the cloud it takes 10+ hours. We have cpu request set to 4 and limit of 20 (this is ...
0
votes
0
answers
35
views
Spring Batch - could not serialize access due to read/write dependencies among transactions
I am facing some issues with Spring Batch metadata tables upon transactions.
org.springframework.dao.CannotAcquireLockException: PreparedStatementCallback; SQL [INSERT INTO ...
0
votes
1
answer
24
views
Spring Batch upgrade - No Scope registered for scope "Step"
I have a spring batch that does not use spring boot. I am upgrading it to Spring batch version 5
The batch has a reader bean which is stepscopped because I need to access the jobparameters in it.
With ...
-1
votes
1
answer
23
views
How to stop the Spring batch job gracefully post app crash or DB restart
I am using spring boot application and spring batch for background job.
When I schedule to job, it is running. Suddenly application restart or DB restart is happening due to some reason.
Now, Spring ...
0
votes
0
answers
17
views
Why caching in springboot not working in multi threaded environment such as spring batch application
I am working on spring batch application and implemented multi-threading (ThreadPoolTaskExecutor) along with using Partition interface provided grid size 12 and number of threads also 12.
For ...
-1
votes
0
answers
29
views
Unexpected Processor Retry Behavior and Fault Tolerance in Spring Batch
@Bean
public Step step1(DataSourceTransactionManager dataSourceTransactionManager) throws Exception {
return new StepBuilder(STEP_NAME, jobRepository)
.<StartTask, ...
2
votes
0
answers
101
views
How to stop the Spring batch gracefully in Spring boot application
I am using Spring batch for my spring boot application. After I schedule the job in the background, i want to stop the job.
I am using following to stop the job
jobOperator.stop(jobId)
Job is going ...
0
votes
1
answer
33
views
How can I use XStream to start parsing from a given XML Path
I have an XML file with the following shape:
<file>
<items>
<item name="item1">
<labels>
<label code="item 2 label 1" />
<...
0
votes
0
answers
14
views
How should long running steps be used in Spring Batch with transaction management?
I'm getting started with Spring Batch and am trying to handle the following scenario:
I have a SQL database with database records in a table. For each of these records, I would like to make an HTTP ...
-1
votes
1
answer
34
views
How do I setup KafkaItemReaderBuilder to read all partitions?
I currently use the partitions method with the list of partitions to be consumed :
partitions(0, 1, 2)
If a partition 3 is added, it will not consume it. How do I tell KafkaItemReaderBuilder I want to ...
0
votes
0
answers
20
views
Not able to consume the message from Kafka topic in latest Spring batch application( Spring Batch 5.x)
We developed a spring boot application using Spring Batch 4 and Spring Cloud Stream 3 earlier. Consume the data from Kafka using @EnableBinding(Sink::class) and @StreamListener(Sink.INPUT) in the ...
0
votes
0
answers
14
views
Creating new Transaction for Spring Batch Writer while handling Error Listener
I have a spring batch job where the Writer gets a list of items. I am using chunk oriented processing. My spring batch chunk size is 100. I have enabled fault tolerant for the step. Since the writer ...
0
votes
2
answers
79
views
Microsoft Graph API fails to update user birthday and hireDate in GCC high environment
I have a spring batch job that is scheduled to update the hireDate and birthday attributes of a user profile in SharePoint using MS Graph API in the GCC high environment. Below are the details:
PATCH ...
0
votes
0
answers
17
views
MongoDB batch document update strategy with Spring Batch
I am trying to update multiple ‘MongoDB’ documents of a collection in TEST database with the same collection in LIVE database using Spring Batch’.
For this purpose I thought to delete the ...
0
votes
1
answer
75
views
Spring Batch. DataSource must not be null
I created a new Spring project to test Spring batch in Java 17. The job is simple, print "Hello, World!".
My batch-job.xml
<?xml version="1.0" encoding="UTF-8"?>...
4
votes
2
answers
177
views
Spring Batch issue with MultiResourceItemWriter and ClassifierCompositeItemWriter
I am using Spring Boot + Batch v2.7.1 in my project and looks like there is a bug when Reading from FlatFileItemReader using ClassifierCompositeItemWriter and MultiResourceItemWriter as ...
0
votes
0
answers
11
views
How to loop a step until a condition in execution context is met?
My usecase is to fetch records from a sharded database and update. i have startstep that sets the counter and increment the loop and not sure how and where to. specify the loop and then decide if the ...
0
votes
1
answer
54
views
How to update existing items during Spring batch migration?
For migration documents between different MongoDB collections, I read items via MongoCursorItemReader based on a filter and then write these data using ItemWriter as shown below. In order to prevent ...
-1
votes
0
answers
16
views
Spring Batch scenario, am I thinking this right?
I want to process millions of accounts. I want to read 1K accounts (pageSize) per DB read, call a REST service in processor with 20 account id's per call, then write to network storage (recommends ...
0
votes
0
answers
21
views
Spring Batch Asynchronous Chunk Processing With Specific Item Grouping
I'm reading messages from the database, grouping them into chunks and passing those chunks to the thread pool for processing. So, each thread picks one chunk and processes it in the scope of one ...
1
vote
1
answer
72
views
How to create dynamic steps in Spring Batch?
I have the following MigrationService and BatchoCnfig classes and want to pass jobParameters to the related steps using MigrationJobListener shown below. However, I am getting "Could not autowire....
0
votes
1
answer
20
views
How can I run jobs in Spring Batch 5.x without creating Batch Metadata entries in Database
As per Spring Batch 5.x, we have to create meta data table in our DB and because of that I am seeing lots of entries in our DB which I am trying to avoid. Is there a way I can avoid job entries in DB ?...
0
votes
2
answers
72
views
How to access JobParameters in Spring Batch ItemReader?
I am trying to access job parameters in an ItemReader as shown below, but when trying some the value cannot be passed or I get some errors e.g.
"Failed to initialize the reader: Cannot determine ...
0
votes
0
answers
36
views
Spring Boot Batch with TaskExecutor OptimisticLockingFailureException
I use Spring Boot 3 (Batch 5) with Spring Data and Oracle Database.
I have one Job with an Item Reader and a Writer (without a processor)
Everything is working fine, Now to make job process fast, want ...
0
votes
0
answers
66
views
Spring Batch 5.1.2: BeanPostProcessor warnings after migrating from @EnableBatchProcessing to DefaultBatchConfiguration
I have a spring batch application (version 5.1.2) and was migrating from the EnableBatchProcessing-Annotation to the programatic configuration by implementing DefaultBatchConfiguration-Class.
I did ...
1
vote
0
answers
22
views
How to persist h2 changes made inside Spring Batch ItemReader
My goal is that while I am processing my files, to also add them in a persistent H2 instance with their current status: "NEW" during reading, "PROCESSING" in the ItemProcessor, and ...
0
votes
0
answers
14
views
How to count the number of files created by composite Item Writer using Spring Batch?
I am using Spring Batch in project and using ClassifierCompositeItemWriter to classify records and then using MultiResourceItemWriterBuilder to create multiple files. In my case more than 1 files is ...
0
votes
0
answers
50
views
Spring batch 5.x JpaPagingItemReader Adding "Top ?" to teradata query
I have a spring batch application that was running fine in spring batch 3.x, then it was upgraded to 5.x and it started to fail with:
Syntax error, expected something like an integer or a decimal ...
0
votes
0
answers
43
views
Unable to create scheduled Tasks in SCDF - 2.11.0
I am using SCDF version 2.11.0 . Earlier I am able to create schedule tasks with CRON_TZ and it was working fine .
Recently , not able to create schedule with CRON_TZ . We are getting the below error ,...
0
votes
1
answer
21
views
How to use declarative transaction management in spring cloud task with spring batch
Spring boot 2.5.4
My problem is that i dont't succeed in managing transactions : while i annotate my service layer and dao with @Transactionnal, when i have successive dao call in the same service ...
0
votes
0
answers
16
views
Dynamic procedure name on StoredProcedureItemReader
I wrote a class that extends StoredProcedureItemReader in the following way:
@Component
public class SpFirstReader extends StoredProcedureItemReader<StudentResults> implements ...
0
votes
0
answers
40
views
Why are Spring converters package protected?
In Spring Batch, job parameters are converted from strings to the declared type, using a JobParametersConverter. The default implementation wraps a (Spring Core) ConversionService, which actually ...
0
votes
0
answers
45
views
CompositeItemWriter having issue while persisting data
I wanted a single reader two processors and two writers. my processing logic is correct but data is not being persisted in DB, this is my composite writer
@Bean
public ...
0
votes
0
answers
20
views
Spring Batch how to create fixed size file when reading multiple flat files?
I'd like to further extend Reading data form multiple csv file and writing it into one csv file using Spring Batch, in my case I've to read multiple files (with the same filename) from different sub-...
0
votes
1
answer
38
views
Spring cloud data flow cannot find driver class at startup
I'm trying to up spring cloud data flow in my local. By default SCDF use h2 as primary database to store meta information. In my cloud-task-batch application all the properties related to database are ...
0
votes
0
answers
37
views
Caused by: org.springframework.jdbc.BadSqlGrammarException: Executing query; bad SQL grammar [SELECT * FROM employee where age in (:ageList)];
I'm using Spring Batch XML code in my project where I'm reading from MySQL DB using IN clause and writing it into CSV/CustomItemWriter etc.
Getting below error, it appears that Spring Batch is not ...
1
vote
2
answers
48
views
Get the Previous Execution time of @Schedule job in springboot
I'm looking for an option to get the last execution time of @Scheduled task. so that I want to use it for retrieving the database records from the last execution time.
Quartz dependency and store the ...
0
votes
0
answers
48
views
How to pass list in SQL query - Spring Batch Caused by: java.sql.SQLException: Parameter index out of range (2 > number of parameters, which is
Before posting the question - I went through below URLS, but it was clear enough to me
Using SpringBatch JdbcCursorItemReader with List as NamedParameters
How to set list of parameters on prepared ...
0
votes
2
answers
35
views
Why I am getting this error? - SLF4J: Class path contains multiple SLF4J bindings
Error :-
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/Users/rm694/.gradle/caches/modules-2/files-2.1/ch.qos.logback/logback-classic/1.2.3/...
0
votes
0
answers
63
views
spring batch 5: how to NOT use the datasource?
I am migrating my spring boot spring batch application from spring batch 4.x to 5.1.1 (this is part of spring boot 2.7.18 to 3.2.4 migration).
In our batch application we do not use any spring batch ...
0
votes
1
answer
61
views
Spring Batch - read date field as a String in readers and convert it to other format
Before posting the question, I went through many links like: Using Spring Batch to parse date from file into LocalDateTime and Spring Batch - how to convert String from file to Date? etc but that ...
0
votes
0
answers
54
views
Loading data from file to database, file has field which is primary key, how to implement hibernate bulk inserts. can't have auto generated Id
Entity looks like this.
@Entity
@Table(name = "TRANSACTION")
@Data
@AllArgsConstructor
@NoArgsConstructor
public class Transaction {
@Id
@Column(name = "ACCOUNT")
private String ...
1
vote
0
answers
69
views
Data Migration of multiple tables from one database to other using Spring Boot and Spring Batch
I want to migrate from one database to other using Spring Batch and Spring Boot. I have learnt that we have to declare @Bean for step, Job, JobLauncher, ItemWriter, ItemReader. But for those beans to ...
1
vote
0
answers
23
views
Spring batch read completion indicator
Requirement:
Read multiple parquet files from a location
Find invalid records, store them in-memory and write them to a file at the end of the batch
Writer writes valid data to the database
Unsure ...
0
votes
1
answer
33
views
Spring Batch - Using ClassifierCompositeItemWriter - how to avoid writting headers when file is empty (no data)
I went through Is there a way to override and not execute the headerCallBack in Spring Batch, but still not clear on the solution. I'm using Spring XML based approach in my project.
I'm using ...
0
votes
0
answers
25
views
Type Mismatch while using FlatFileItemReader BeanWrapperFieldSetMapper from String to Date
I am trying to batch process a file which has date field. Now when I fetch Date field using FlatFileItemReader it comes as a String.
I am trying to map it to following entity
@Entity
@Data
public ...
0
votes
0
answers
19
views
Spring Batch - Using Custom Classifier how to create two more files
I'm using Spring Batch XML for my project and have below requirement to fullfill. My task is
connect Sybase DB, read all records from the table
Based on clientId (there are 10 different clientId), ...