DSXchange

Hi.. i have job design in server jobs......where i have to convert number of rows to no of columns.....

i have designed a job by using pivot stage, where i will get 30,00,000 Records from source.

Now this job is running for 30mins..Please help me to increase the performance.

Please see the sample recodrs below..


Source:

Sysdate,HK,ODS,100,200,300,400,500,a,b,c,d,e,1,2,3,4,5,XXX

Target:

Sysdate HK ODS 100 a 1 XXX
Sysdate HK ODS 200 b 2 XXX
Sysdate HK ODS 300 c 3 XXX
Sysdate HK ODS 400 d 4 XXX
Sysdate HK ODS 500 e 5 XXX

What else is happening in the job? What are your source and target? Where is the alleged slowness? How can you be sure -- have you performed any analysis? Are there any transformations or other computations being carried out? Depending on the total workload combined with everything else happening on the machine it may be the case that 30 minutes is the best you're going to get.

That's still one million rows per minute, or 1667 rows per second, if you believe that rows/sec is a meaningful metric.

In the job design source is squential file and target is Oracle database and also Tranformer stage in between.


In the design at pivot stage it's taking time to convert the number of rows to number of column.

Below are the sequence of stage

sequential File stage, Transformer Stage(For to do null handle), Pivot stage and Oracle Stage.

try to substitute oracle-target with sequential target and look if it is quicker and give us the answer.

pdv wrote:In the design at pivot stage it's taking time to convert the number of rows to number of column.

What proof do you have of this assertion?