One stage is sequential in parallel jon

vishu19aug · Post by **vishu19aug** » Thu Feb 23, 2012 9:17 am

hi,

In my parallel job one stage (transformer) is sequential. Does it have any negative impact on performance? will it be bottleneck for the complete prallel flow?

Thanks,
Vishal

mandyli · Post by **mandyli** » Thu Feb 23, 2012 9:31 am

Hi

Have you developed and tested with any data ?

without knowing anything how we will say about performance.

Is this is simple and low number of rows or huge?

Thanks
Man

vishu19aug · Post by **vishu19aug** » Thu Feb 23, 2012 11:06 am

Yes i did but with very small data. However, my job will process around 1 million records.

jwiles · Post by **jwiles** » Thu Feb 23, 2012 12:39 pm

Is the transformer sequential for a certain reason? Does it HAVE to be sequential in order to meet business logic requirements?

Any sequentially operating stage in a parallel job has the potential to be a bottleneck depending upon the job design and data being processed, but that doesn't mean they will be.

Regards,

vishu19aug · Post by **vishu19aug** » Thu Feb 23, 2012 2:12 pm

I have the following logic to compare the name of two records. Where the file is sorted on name and rownumber, because there can be two groups same name but sandwitched with another group of name. THe below does not work correctly if the stage is parallel

StageVar1 = Input.name
StageVar2 = (if StageVar1 = StageVar3 then StageVar4 else StageVar4 + 1)
StageVar3 = StageVar1
StageVar4 = StageVar2

ray.wurlod · Post by **ray.wurlod** » Thu Feb 23, 2012 3:06 pm

Get the partitioning right and it WILL work properly in parallel. A sequential stage in a parallel job will always be a bottleneck.