Wednesday, 14 August 2013

Tasklet with multi threaded executor has bad performances

Tasklet with multi threaded executor has bad performances

I have configured a Spring Batch tasklet with:
1 jdbc ItemReader, with a synchronized adapter to allow it's use by many
threads, saveState is false
1 Custom ItemWriter which does processing which can vary in terms of
processing time, it ca take few millis to few seconds ( > 60s) per item
no retry configured
Chunk with pooled executor limited to 15 threads
commit-limit set to 1 ( it's not the issue)
throttle is equal to number of threads
All jdbc pools are fine
Spring batch 2.2.1 is used
I used this approach:
http://static.springsource.org/spring-batch/reference/html/scalability.html#multithreadedStep
Running the batch leads to very strange and bad results due to the
following, at some step, if the items take some time to process. by a
writer, the thread pool ends up doing nothing instead of processing.
Analyzing code, it comes from the queue used for throttling which blocks
in tasklet thread waiting the queue to be filled by writer so that it
notifies take() method.
My. question is is there a way to tweak by configuration , so that all my
threads always work and dont get starved by long processing ?
Or should I have not used tasklet and chunk with parallel processing ?

No comments:

Post a Comment