to make the Transformer Wait for the file

sshettar · Post by **sshettar** » Thu Dec 14, 2006 3:08 pm

Well i have this job where in i need to make the transformer wait for the file that is written in the same same job .
the req i have is that i need the count of the records getting into the output file and at the same time i need to extract the input record count from the input file and load both of these info into a single Oracle table .all these are done from the transfprmer stage.
the problem i'm facing is that the transformer is extraxting the input record count real quick ie much bofore all the records are loaded into the output file . so i'm not getting the right output count as the transformer loads the input count and the output count as per the number of records passed at that instant ,into the oracle table the moment it completes getting the input count .
Can somebody help me with this issue ?

Thanks
Shalini

I_Server_Whale · Post by **I_Server_Whale** » Thu Dec 14, 2006 3:57 pm

I don't think you could achieve your requirement with your kind of job design.

wait_for_file has a different meaning, Usually, a process waits for a file if its not present at the source location to read from. In your case, you already have the input file.

Also, you cannot read and write into the same sequential file at the same time.

You will get a

Code: Select all


ds_seqopen() - Win32 error in CreateFile - The process cannot access the file because it is being used by another process.

error.

Whale.

us1aslam1us · Post by **us1aslam1us** » Thu Dec 14, 2006 4:08 pm

Can you share your job design with us?

As whale had suggested it is not possible to read and write to a file at the same time but i think that you are not doing this.

Sam

narasimha · Post by **narasimha** » Thu Dec 14, 2006 4:15 pm

Check if this post helps you.

thebird · Post by **thebird** » Thu Dec 14, 2006 4:25 pm

How are you getting to the input and output count in your job?

If you need the number of records written into the file, you would need a separate job to load this information into the table. If it is the number of records going into the file (assuming that you dont want the reject count if any) - then you can catch it with an aggregator before writing into the file.

kris007 · Post by **kris007** » Thu Dec 14, 2006 4:26 pm

I don't think you can achieve this in a single job. However, what you could do is capture INROWNUM and OUTROWNUM in a different file through a different link from the transformer stage and then use this file in another job to load the table.

DSguru2B · Post by **DSguru2B** » Thu Dec 14, 2006 7:46 pm

Modularization my friend. Build the text files in one job. Cat them and load in the second job.

ray.wurlod · Post by **ray.wurlod** » Thu Dec 14, 2006 7:59 pm

This design will not open the file for reading until it has been written to and closed by the writer, therefore the Transformer stage will wait.

Code: Select all

----->  SeqFileStage  ----->  TransformerStage  ----->

But a modular design in which one job creates the file and another reads it, is easier to maintain. And the second job can be controlled by a WaitForFile activity in a job sequence.