Page 1 of 1
Is there any limit on the number of rows in transformer
Posted: Tue Nov 23, 2004 4:07 am
by DEVESHASTHANA
Hi,
Is there any limit on the number of rows in transformer in Datastage ?
If yes then what is the limit?
Thanking u in anticipation.
regards,
Devesh
Re: Is there any limit on the number of rows in transformer
Posted: Tue Nov 23, 2004 5:23 am
by denzilsyb
DEVESHASTHANA wrote:Is there any limit on the number of rows in transformer in Datastage ?
you mean columns? i dont know what the limit is, but i have tried to go beyond 1 thousand and designer died.
how many columns do you want to work with? We have worked with just under 200 quite safely.
Posted: Mon Nov 29, 2004 5:44 am
by DEVESHASTHANA
Hi,
In one of the Fact job i am using 274 columns and desingner goes to infinite loop(hangs) ,i dont know the reason that might be because of transformer limit ,or it might be because of 67 lookups i am using for that fact.
regards,
Devesh
Posted: Mon Nov 29, 2004 5:56 am
by vigneshra
67 lookups in a single job ?? That sounds problematic...You break your job to simpler sub-jobs using filesets and datasets and that is what they are meant for.. Instead of one complex job, break it into a few simpler jobs and infact, this would yield better performance to you.
Vignesh.
Posted: Fri Dec 10, 2004 1:09 pm
by DEVESHASTHANA
hi vignesh ,
Thanks,
What is this file set and data set u are talking about can u please elaborate this point,
regards,
Devesh
Posted: Fri Dec 10, 2004 1:35 pm
by ogmios
Data Sets is something of the PX edition of DataStage (now Enterprise Edition).
For your problem every lookup you use launches an extra query to the database so 67 lookups means that for every fact row at least 67 queries are executed, probably your job is not hanging but just very very very slow.
Normally I use at maximum 1 or 2 lookups in a job. Can't you do the joins you want to do in the database (in the input query). If you do about 20 joins in a job you can get away by making 3 DataStage jobs.
Ogmios
Posted: Fri Dec 10, 2004 2:14 pm
by ray.wurlod
There are two easy ways to limit the number of rows processed by a Transformer stage.
One is to add constraints to the output link(s), for example
The other is to open the Constraints editor grid. The rightmost column of this grid allows you to specify a number of rows on each output link. As soon as this limit is exceeded, the job aborts.
The first approach is cleaner.
The default limit is 0 (unlimited).