Spring Batch (Michael Minella)

What people said

Recommends Michael Minella · 2013-10 · Stack Overflow · on “Spring batch- Parallel processing”

What you are describing is partitioning. Partitioning is when the input is broken up into partitions and each partition is processed in parallel. Spring Batch offers two different ways to execute partitioning, one is local using threads (via the TaskExecutorPartitionHandler). The other one is distributing the partitions via messages so they can be executed either locally or remotely via the MessageChannelPartitionHandler found in Spring Batch Admin's spring-batch-integration project. You can learn more about remote partitioning via my talk on multi-jvm batch processing here: [this video]

Recommends Michael Minella · 2013-12 · Stack Overflow · on “Spring Batch: Which ItemReader implementation to use for high volume & low latency”

To process that kind of data, you're probably going to want to parallelize it if that is possible (the only thing preventing it would be if the output file needed to retain an order from the input). Assuming you are going to parallelize your processing, you are then left with two main options for this type of use case (from what you have provided): Multithreaded step - This will process a chunk per thread until complete. This allows for parallelization in a very easy way (simply adding a TaskExecutor to your step definition). With this, you do loose restartability out of the box because you will need to turn off state persistence on either of the ItemReaders you have mentioned (there are way

Recommends Michael Minella · 2014-06 · Stack Overflow · on “Configure spring batch partition to for processor and writer whilst excluding the reader from the partitioning”

Without the configuration for your reader, it's tough to give you an exact fix, but my bet is that you are not injecting the range values into your query. Below is an example of using the JdbcPagingItemReader in a remote partitioning job. You'll notice that the range of items being read are provided by the stepExecutionContext . Those are the values provided by the partitioner. Each ItemReader will get its own values. <bean id="targetItemReader" class="org.springframework.batch.item.database.JdbcPagingItemReader" scope="step"> <property name="dataSource" ref="dataSource" /> <property name="queryProvider"> <bean class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFact

Recommends Michael Minella · 2014-07 · Stack Overflow · on “Spring Batch - Reading a large flat file - Choices to scale horizontally?”

Assuming "minor processing" isn't the bottle neck in the processing, the best option to scale this type of job is via partitioning. The job would have two steps. The first would split the large file into smaller files. To do this, I'd recommend using the SystemCommandTasklet to shell out to the OS to split the file (this is typically more performant than streaming the entire file through the JVM). An example of doing that would look something like this: <bean id="fileSplittingTasklet" class="org.springframework.batch.core.step.tasklet.SystemCommandTasklet" scope="step"> <property name="command" value="split -a 5 -l 10000 #{jobParameters['inputFile']} #{jobParameters['stagingDirectory']}"/> <

Recommends Michael Minella · 2015-04 · Stack Overflow · on “spring batch remote partition with multiple tables”

Yes this can be done. You'll need to implement your own Partitioner to handle the logic of creating ExecutionContexts for each table/row range. A couple good resources: Partitioner interface documentation: [LINK] Section 7.4 of Spring Batch's Scaling and Parallel Processing chapter: [LINK] My introduction to Spring Batch video covers partitioning: [this video] and the related Github repository: [LINK]

Cites Michael Minella · 2013-09 · Stack Overflow · on “Making Existing Spring Batch Application run on multiple nodes”

It really depends on your situation. Specifically, why do you do you want to run on multiple nodes? What is the bottle neck you're attempting to overcome? The typical two scenarios that Spring Batch handles out of the box for scaling across multiple nodes are remote chunking and remote partitioning. Both are master/slave configurations, but each have a different use case. Remote chunking is used when the processor in a step is the bottle neck. In this case, the master node reads the input and sends it via a Spring Integration channel to remote nodes for processing. Once the item has been processed, the result is returned to the master for writing. In this case, reading and writing are done l

Cites Michael Minella · 2013-12 · Stack Overflow · on “Difference between spring batch remote chunking and remote partitioning”

Remote Partitioning Partitioning is a master/worker step configuration that allows for partitions of data to be processed in parallel. Each partition is described via some metadata. For example, if you were processing a database table, partition 1 may be ids 0-100, partition 2 being 101-200, etc. For Spring Batch, a master step uses a Partitioner [LINK] to generate ExecutionContexts that contain the metadata for each partition. These ExecutionContexts are distributed to worker steps for processing by a PartitionHandler [LINK] (for remote partitioning, the MessageChannelPartitionHandler [LINK] is typically used). The workers execute their step and return the resulting statuses for aggregation

Cites Michael Minella · 2015-07 · Stack Overflow · on “Using Spring Batch to horizontally scale external process execution”

What you are describing is exactly what partitioning does. Even your "out of band" option still falls into what partitioning does. Let's walk through what I would expect the job to look like. Job and Master Step The job, as you noted, is a single step job. What I would envision is that the single step is a partitioned step. With a partitioned step, the two main pieces you need to configure are the Partitioner (the component that knows how to divide the work up) and the PartitionHandler (the component that knows how to send the work to the workers). For the Partitioner , I'd expect using the MultiResourcePartitioner [LINK] would work. This Partitioner implementation provided by Spring Batch c

Cites nanpakal · 2016-02 · Stack Overflow · on “How to start slaves on different machines in spring remote partitioning strategy”

I am using spring batch local partitioning to process my Job.In local partitioning multiple slaves will be created in same instance i.e in the same job. How Remote partitioning is different from local partitioning.What i am assuming is that in Remote partitioning each slave will be executed in different machine. Is my understanding correct. If my understanding is correct how to start the slaves in different machines without using cloudfoundry. I have seen Michael Minella talk on Remote partitioning [this video] tutorial. I am curious to know how remote partitioning works without using cloudfoundry. How can I start slaves in different machines?

Cites pojo-guy · 2019-04 · Stack Overflow · on “Spring Batch integration - How to get the remote steps that are external to the job visible to the StepLocator on the remote execution JVM?”

I've got my jobs configured for remote partitioning, I'm sending the message through kafka, the slave VM is receiving the requests and attempting to kick off the step, but it can't find the step. It appears that the beanFactory (applicationContext) I am providing does not contain the remote execution step. I have walked through the example at [this video] several times, and am missing how the correct beanFactory is identified and set for the slave jobs. During application initialization, the ClasspathXmlApplicationContextsFactoryBean is loaded from the jobs library, which contains XML for all of the spring batch jobs and the remote slave steps. This bean contains a separate resource for each XML