Is there any limit on the number of rows in transformer

[DEVESHASTHANA]

Hi,
Is there any limit on the number of rows in transformer in Datastage ?

If yes then what is the limit?

Thanking u in anticipation.

regards,
Devesh

[denzilsyb]

Is there any limit on the number of rows in transformer in Datastage ?

you mean columns? i dont know what the limit is, but i have tried to go beyond 1 thousand and designer died.

how many columns do you want to work with? We have worked with just under 200 quite safely.

[DEVESHASTHANA]

Hi,

In one of the Fact job i am using 274 columns and desingner goes to infinite loop(hangs) ,i dont know the reason that might be because of transformer limit ,or it might be because of 67 lookups i am using for that fact.


regards,

Devesh

[vigneshra]

67 lookups in a single job ?? That sounds problematic...You break your job to simpler sub-jobs using filesets and datasets and that is what they are meant for.. Instead of one complex job, break it into a few simpler jobs and infact, this would yield better performance to you.

Vignesh.

[DEVESHASTHANA]

hi vignesh ,
Thanks,

What is this file set and data set u are talking about can u please elaborate this point,

regards,
Devesh

[ogmios]

Data Sets is something of the PX edition of DataStage (now Enterprise Edition).

For your problem every lookup you use launches an extra query to the database so 67 lookups means that for every fact row at least 67 queries are executed, probably your job is not hanging but just very very very slow.

Normally I use at maximum 1 or 2 lookups in a job. Can't you do the joins you want to do in the database (in the input query). If you do about 20 joins in a job you can get away by making 3 DataStage jobs.

Ogmios

[ray.wurlod]

There are two easy ways to limit the number of rows processed by a Transformer stage.

One is to add constraints to the output link(s), for example

Code: Select all

@INROWNUM < 100

The other is to open the Constraints editor grid. The rightmost column of this grid allows you to specify a number of rows on each output link. As soon as this limit is exceeded, the job aborts.

The first approach is cleaner.

The default limit is 0 (unlimited).