how to read multiple stages in datastage one by one

hellboy · Post by **hellboy** » Fri Mar 26, 2010 8:01 pm

how to read multiple stages in datastage one by one

my req is like i have 3 input file stages with me , where i want to start reading the second input stage only when the 1st get over same for the 3rd when 2nd get over...

is this possible in data stage to di it in same job.

chulett · Post by **chulett** » Fri Mar 26, 2010 8:51 pm

No. It might help to know what's in these three input stages, however, to determine a best course of action for you.

hellboy · Post by **hellboy** » Fri Mar 26, 2010 10:38 pm

input stages are seqfile stages and the data is normal business data ,i just want to check wheater it is possible to read the files one by one .
when ever i m trying to do the testing every time stages start reading on same time ... do i need to setup any constarints to achieve this result.

chulett · Post by **chulett** » Fri Mar 26, 2010 10:49 pm

No, they will always run in parallel. Again, why three separate stages? Is the metadata different? Can you not read all three in a single stage using a File Pattern?

ray.wurlod · Post by **ray.wurlod** » Sat Mar 27, 2010 12:55 am

Even that may be in parallel.

You need a single File property and a Filter Command to cat the three files together.

chulett · Post by **chulett** » Sat Mar 27, 2010 7:51 am

Assuming you can. We still don't know the reasoning behind 'needing' three separate stages or the serial processing.

keshav0307 · Post by **keshav0307** » Sun Mar 28, 2010 8:12 pm

Assuming you need to do some data validation of each file, and that's the reason you can't combined 3 files into 1 (using cat command etc.) .

you can have a single job with multi instance, to append into a single file, from the three(or more) instance of job.

hellboy · Post by **hellboy** » Sun Mar 28, 2010 10:23 pm

thanks for the valuable suggestions....

no the input meta is different but i need to combine the output in a single file with the common columns from the input files.

so is it not possible in a single job to read one after the other...???

keshav0307 · Post by **keshav0307** » Sun Mar 28, 2010 11:05 pm

Server job it is certainly possible and easy to implement.

Parallel , possible, tricky. and probably not very useful.

i don't have PX environment now so can't test.
try this

take the last records from 1st file as refrence input to the second file using lookup stage, so that the second file will wait untill it get the last record from 1st file.

hellboy · Post by **hellboy** » Sun Mar 28, 2010 11:24 pm

thanks keshav,
yes u r right by this way we can go on...

can u please tell us how it is possible in Server job....

ray.wurlod · Post by **ray.wurlod** » Mon Mar 29, 2010 12:17 am

U is not posting in this forum today. And certainly not in this thread. (Though I note that U has made a post today.)

The second person personal pronoun in English is spelled "you". DSXchange is not a mobile telephone nor is there any requirement to limit your messages to 72 or 144 characters. Please maintain a professional standard of written English on DSXchange, not least because it helps those whose first language is not English.

keshav0307 · Post by **keshav0307** » Mon Mar 29, 2010 12:22 am

Read 1st file, and write into a sequential file(file_1), and in the output of the file_1 read the 2nd file, and append to file_1 , again in the output read 3rd file and append to File_1

hellboy · Post by **hellboy** » Mon Mar 29, 2010 12:23 am

Thanks keshav,

Yes you are right by this way we can go on...

Could you please tell us how it is possible in Server job....?

thanks for the guiding me...

ray.wurlod · Post by **ray.wurlod** » Mon Mar 29, 2010 12:24 am

ray.wurlod · Post by **ray.wurlod** » Mon Mar 29, 2010 12:26 am

hellboy wrote:Could you please tell us how it is possible in Server job....?

Server job questions really belong in the server job forum.

This helps future searchers.