7

Context

I'm trying to develop a batch service with Spring Boot, using JPA Repository. Using two different datasources, I want the batch-related tables created in a in-memory database, so that it does not pollute my business database.

Following multiple topics on the web, I came up with this configuration of my two datasources :

@Configuration
public class DataSourceConfiguration {

    @Bean(name = "mainDataSource")
    @Primary
    @ConfigurationProperties(prefix="spring.datasource")
    public DataSource mainDataSource(){
        return DataSourceBuilder.create().build();
    }

    @Bean(name = "batchDataSource")
    public DataSource batchDataSource( @Value("${batch.datasource.url}") String url ){
        return DataSourceBuilder.create().url( url ).build();
    }   
}

The first one, mainDataSource, uses the default Spring database configuration. The batchDataSource defines an embedded HSQL database, in which I want the batch and step tables to be created.

# DATASOURCE (DataSourceAutoConfiguration & DataSourceProperties)
spring.datasource.url=jdbc:mariadb://localhost:3306/batch_poc
spring.datasource.username=root
spring.datasource.password=
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.max-age=10000

spring.datasource.initialize=false

# JPA (JpaBaseConfiguration, HibernateJpaAutoConfiguration)
spring.jpa.generate-ddl=false
spring.jpa.show-sql=true
spring.jpa.database=MYSQL

# SPRING BATCH (BatchDatabaseInitializer)
spring.batch.initializer.enabled=false

# ----------------------------------------
# PROJECT SPECIFIC PROPERTIES
# ----------------------------------------

# BATCH DATASOURCE
batch.datasource.url=jdbc:hsqldb:file:C:/tmp/hsqldb/batchdb

Here is my batch config :

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    private static final Logger LOG = Logger.getLogger( BatchConfiguration.class );

    @Bean
    public BatchConfigurer configurer(){
        return new CustomBatchConfigurer();
    }

    @Bean
    public Job importElementsJob( JobBuilderFactory jobs, Step step1 ){
        return jobs.get("importElementsJob")
                .incrementer( new RunIdIncrementer() )
                .flow( step1 )
                .end()
                .build();               
    }

    @Bean
    public Step step1( StepBuilderFactory stepBuilderFactory, ItemReader<InputElement> reader,
            ItemWriter<List<Entity>> writer, ItemProcessor<InputElement, List<Entity>> processor ){

        return stepBuilderFactory.get("step1")
                .<InputElement, List<Entity>> chunk(100)
                .reader( reader )
                .processor( processor )
                .writer( writer )
                .build();
    }

    @Bean
    public ItemReader<InputElement> reader() throws IOException {       
        return new CustomItemReader();
    }

    @Bean
    public ItemProcessor<InputElement, List<Entity>> processor(){
        return new CutsomItemProcessor();
    }

    @Bean
    public ItemWriter<List<Entity>> writer(){
        return new CustomItemWriter();
    }

}

The BatchConfigurer, using the in-memory database :

public class CustomBatchConfigurer extends DefaultBatchConfigurer {

    @Override
    @Autowired
    public void setDataSource( @Qualifier("batchDataSource") DataSource dataSource) {
        super.setDataSource(dataSource);
    }

}

And, finally, my writer :

public class CustomItemWriter implements ItemWriter<List<Entity>> {

    private static final Logger LOG = Logger.getLogger( EntityWriter.class );

    @Autowired
    private EntityRepository entityRepository;

    @Override
    public void write(List<? extends List<Entity>> items)
            throws Exception {
        if( items != null && !items.isEmpty() ){

            for( List<Entity> entities : items ){
                for( Entity entity : entities ){                        
                    Entity fromDb = entityRepository.findById( entity.getId() );

                    // Insert
                    if( fromDb == null ){
                        entityRepository.save( entity );
                    }

                    // Update
                    else {
                        // TODO : entityManager.merge()
                    }
                }
            }

        }
    }

}

The EntityRepository interface extends JpaRepository.

Problem

When I separate the datasources this way, nothing happens when I call the save method of the repository. I see the select queries from the call of findById() in the logs. But nothing for the save. And my output database is empty at the end.

When I come back to a unique datasource configuration (removing the configurer bean and letting Spring Boot manage the datasource alone), the insert queries work fine.

Maybe the main datasource configuration is not good enough for JPA to perform the inserts correctly. But what is missing ?

2
  • 2
    Since you are using two different datasources, could you have a transaction issue? Are both Datasources registered with the same TransactionManager? per default, this is the bean named "transactionManager" in the context. SpringBatch commits this TM and if your Datasource isn't registered there, it will never receive a commit. Oct 13, 2015 at 10:11
  • 1
    I think I had a transactionManager issue, yes. With the solution posted below, I define myself a JpaTransactionManager with the EntityManagerFactory bean created by Spring, and it works.
    – Eria
    Oct 13, 2015 at 12:02

3 Answers 3

14

I finally solved the problem implementing my own BatchConfigurer, on the base of the Spring class BasicBatchConfigurer, and forcing the use of Map based jobRepository and jobExplorer. No more custom datasource configuration, only one datasource which I let Spring Boot manage : it's easier that way.

My custom BatchConfigurer :

public class CustomBatchConfigurer implements BatchConfigurer {

    private static final Logger LOG = Logger.getLogger( CustomBatchConfigurer.class );

    private final EntityManagerFactory entityManagerFactory;

    private PlatformTransactionManager transactionManager;

    private JobRepository jobRepository;

    private JobLauncher jobLauncher;

    private JobExplorer jobExplorer;

    /**
     * Create a new {@link CustomBatchConfigurer} instance.
     * @param entityManagerFactory the entity manager factory
     */
    public CustomBatchConfigurer( EntityManagerFactory entityManagerFactory ) {
        this.entityManagerFactory = entityManagerFactory;
    }

    @Override
    public JobRepository getJobRepository() {
        return this.jobRepository;
    }

    @Override
    public PlatformTransactionManager getTransactionManager() {
        return this.transactionManager;
    }

    @Override
    public JobLauncher getJobLauncher() {
        return this.jobLauncher;
    }

    @Override
    public JobExplorer getJobExplorer() throws Exception {
        return this.jobExplorer;
    }

    @PostConstruct
    public void initialize() {
        try {
            // transactionManager:
            LOG.info("Forcing the use of a JPA transactionManager");
            if( this.entityManagerFactory == null ){
                throw new Exception("Unable to initialize batch configurer : entityManagerFactory must not be null");
            }
            this.transactionManager = new JpaTransactionManager( this.entityManagerFactory );

            // jobRepository:
            LOG.info("Forcing the use of a Map based JobRepository");
            MapJobRepositoryFactoryBean jobRepositoryFactory = new MapJobRepositoryFactoryBean( this.transactionManager );
            jobRepositoryFactory.afterPropertiesSet();
            this.jobRepository = jobRepositoryFactory.getObject();

            // jobLauncher:
            SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
            jobLauncher.setJobRepository(getJobRepository());
            jobLauncher.afterPropertiesSet();
            this.jobLauncher = jobLauncher;

            // jobExplorer:
            MapJobExplorerFactoryBean jobExplorerFactory = new MapJobExplorerFactoryBean(jobRepositoryFactory);
            jobExplorerFactory.afterPropertiesSet();
            this.jobExplorer = jobExplorerFactory.getObject();
        }
        catch (Exception ex) {
            throw new IllegalStateException("Unable to initialize Spring Batch", ex);
        }
    }

}

My configuration class looks like this now :

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Bean
    public BatchConfigurer configurer( EntityManagerFactory entityManagerFactory ){
        return new CustomBatchConfigurer( entityManagerFactory );
    }

    [...]

}

And my properties files :

# DATASOURCE (DataSourceAutoConfiguration & DataSourceProperties)
spring.datasource.url=jdbc:mariadb://localhost:3306/inotr_poc
spring.datasource.username=root
spring.datasource.password=root
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.max-age=10000

spring.datasource.initialize=true

# JPA (JpaBaseConfiguration, HibernateJpaAutoConfiguration)
spring.jpa.generate-ddl=false
spring.jpa.show-sql=true
spring.jpa.database=MYSQL

# SPRING BATCH (BatchDatabaseInitializer)
spring.batch.initializer.enabled=false
2
3

Thanks for the above posts! I have been struggling for the past couple of days to get my Spring Boot with Batch application working with an in-memory Map based job repository and a resourceless transaction manager. ( I cannot let Spring Batch to use my application datasource for the Batch meta data tables as I don't have the DDL access to create the BATCH_ tables there ) Finally arrived at the below configuration after looking at the above posts and it worked perfectly!!

public class CustomBatchConfigurer implements BatchConfigurer {

    private static final Logger LOG = LoggerFactory.getLogger(CustomBatchConfigurer.class);

   // private final EntityManagerFactory entityManagerFactory;

    private PlatformTransactionManager transactionManager;

    private JobRepository jobRepository;

    private JobLauncher jobLauncher;

    private JobExplorer jobExplorer;

    /**
     * Create a new {@link CustomBatchConfigurer} instance.
     * @param entityManagerFactory the entity manager factory

    public CustomBatchConfigurer( EntityManagerFactory entityManagerFactory ) {
        this.entityManagerFactory = entityManagerFactory;
    }
*/
    @Override
    public JobRepository getJobRepository() {
        return this.jobRepository;
    }

    @Override
    public PlatformTransactionManager getTransactionManager() {
        return this.transactionManager;
    }

    @Override
    public JobLauncher getJobLauncher() {
        return this.jobLauncher;
    }

    @Override
    public JobExplorer getJobExplorer() throws Exception {
        return this.jobExplorer;
    }

    @PostConstruct
    public void initialize() {
        try {
            // transactionManager:
            LOG.info("Forcing the use of a Resourceless transactionManager");

            this.transactionManager = new ResourcelessTransactionManager();

            // jobRepository:
            LOG.info("Forcing the use of a Map based JobRepository");
            MapJobRepositoryFactoryBean jobRepositoryFactory = new MapJobRepositoryFactoryBean( this.transactionManager );
            jobRepositoryFactory.afterPropertiesSet();
            this.jobRepository = jobRepositoryFactory.getObject();

            // jobLauncher:
            SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
            jobLauncher.setJobRepository(getJobRepository());
            jobLauncher.afterPropertiesSet();
            this.jobLauncher = jobLauncher;

            // jobExplorer:
            MapJobExplorerFactoryBean jobExplorerFactory = new MapJobExplorerFactoryBean(jobRepositoryFactory);
            jobExplorerFactory.afterPropertiesSet();
            this.jobExplorer = jobExplorerFactory.getObject();
        }
        catch (Exception ex) {
            throw new IllegalStateException("Unable to initialize Spring Batch", ex);
        }
    }

}

And below is the bean i added in my job configuration class

@Bean
public BatchConfigurer configurer(){
    return new CustomBatchConfigurer();
}
1
  • 3
    with this configuration, spring data jpa save is not working. Unable to save anything in DB. Did you face this issue. Mar 24, 2020 at 3:57
0

Eria's answer worked! However, I had modified it to use:

org.springframework.batch.support.transaction.ResourcelessTransactionManager

From CustomBatchConfigurer:

    @PostConstruct
    public void initialize() {
        try {
            // transactionManager:
            LOGGER.info("Forcing the use of ResourcelessTransactionManager for batch db");

            this.transactionManager = new ResourcelessTransactionManager(); 
            //the rest of the code follows... 
    }

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.