OIC - Parallel Processing

The other day I stumbled upon a check box that enables Parallel Processing. This is part of the for-each action.

According to the documentation , "If you drag a for-each action into a scheduled integration ..., an additional field called Process items in parallel is visible at the bottom of the dialog."

I have tested this and it does provide a considerable performance boost :)

So let's put this into action. I have a simple Scheduled Integration that reads a file from an FTP location, performs a loop that for each line it will publish the record to a Kafka Topic.
The writing sequence is irrelevant so we don't need to ensure an order - it's the perfect use case for parallel processing. If we are writing a File where the order is important, then we should avoid this option.

So, first things first - Lets run this with the default option

Everything you ever wanted to know about the For-Each action can be found here.

The file that the Integration picks up from the FTP location has 2k lines, which will result in 2k published records to the Kafka Topic. 

This run resulted in 5 min and 12 seconds of duration.

Lets go back and change the parallel processing option.

The expectation is that the processing time will be shorter.

And indeed - It roughly cut down execution time in half, which is quite good for such a simple checkbox option :)

There is an obvious limit here which is the number of available threads, but this seems to work on a best effort basis, so nothing to worry about - no errors will be throw, the system takes care of it! 

There are also other design patterns that enable parallel processing, like a parent Integration calling several child Integrations, but that is for another post.