Sequential File stage

dsdeveloper123 · Post by **dsdeveloper123** » Wed Jul 11, 2007 7:08 am

Hi..

I have a scenario where in I have to reading 4 files and transform them and then load it into a single file.

My problem is the job loads only data from one file. It doesn't throw any error but rather than loading 30 records each from 4 files it loads only data from one file. I have checked append to file option also.

Can anyone help me locate where I am going wrong.

chulett · Post by **chulett** » Wed Jul 11, 2007 7:27 am

How, exactly, are you "reading 4 files" in your Server job?

dsdeveloper123 · Post by **dsdeveloper123** » Wed Jul 11, 2007 7:29 am

I am using sequential file stage to read as well as to write the file

chulett · Post by **chulett** » Wed Jul 11, 2007 7:32 am

Not nearly enough information.

How did you setup the stage to 'read 4 files'?

Maveric · Post by **Maveric** » Wed Jul 11, 2007 7:46 am

Is there any similarity between the file names? If so then you can probably use a wild character to read all the files. Example f1.txt, f2.txt, f3.txt. then you can use f*.txt. But for this the meta data for the files should be the same. Just make sure there are no other files in the folder with the same name pattern.
"I have checked append to file option also" This is in the target. but your problem is in the source stage. Try this and see if it will work.

dsdeveloper123 · Post by **dsdeveloper123** » Wed Jul 11, 2007 7:53 am

sorry ..

I have four sequential file to read data from 4 files. Each file I apply different transformation and output from the transformer is sent to a single file stage.

4 sequential file stage(Input) --> 4 transformer stage (for each file)--->1 sequential file stage(output)

My problem is my final output sequential file has data only from the one file.

Hope I am clear now

chulett · Post by **chulett** » Wed Jul 11, 2007 8:07 am

That's what I was afraid of. You need to understand that sequential files need to be accessed sequentially, it's the nature of the beast. This means while a sequential file can support multiple reader processes, it can only support a single writer process.

This is not a 'DataStage restriction', it is a sequential file restriction regardless of platform or application. You may end up with one file's worth or a mixture in your output file when all is done, and the setting of the 'Append' option doesn't help here at all.

So, you'll either need to process the four files one at a time and append each to the previous one or write to four different files and then concatenate the results together 'after job' to build your final file.

ps. The latter is a very typical solution to this 'problem' and is the easiest to arrange.

Maveric · Post by **Maveric** » Wed Jul 11, 2007 8:12 am

Put in a link collector before the sequential file stage. collect the data and then write it into the sequential file.

chulett · Post by **chulett** » Wed Jul 11, 2007 8:31 am

That can work as well. I tend to forget out the 'Link' stages, the Collector will allow the creation of your single writer process. I find the extra overhead of needing to use the 'Sort/Merge' option if you care about the output order to not be worth it, however.

But that's a personal preference. Use whatever works for you.

dsdeveloper123 · Post by **dsdeveloper123** » Wed Jul 11, 2007 8:39 am

Thank u chulett ..
I had not thought in this direction.

I tried using Link collector but I get a compilation error

"Link Collector stage does not support in-process active-to-active inputs or outputs"

Minhajuddin · Post by **Minhajuddin** » Wed Jul 11, 2007 8:50 am

Click on the job properties and..

In the Job properties ........

Uncheck the Project defaults checkbox

And click on the second radio button.. (I think it's Inter process)

Then the compilation warning should go away.

dsdeveloper123 · Post by **dsdeveloper123** » Wed Jul 11, 2007 9:01 am

Thank You all..

It finally worked.. :D
I did as u suggested