multiple inputs to flat file.

[babbu9]

Hi
I am trying to capture the rejects of all of my jobs into a flat file.I overwrite the existing file in the first job feeding my flat file and thereafter append records to the flat file from all the other jobs in the sequence. I would like to know whether this aproach would work.

If not what other way can be implemented so that I can collect all the rejects into a single flat file.

Please inform.

Bob

[ArndW]

There is another live thread on this same subject bouncing around. Basically sequential files are 1 writer and-or N-readers, so you can't have more than one process write to such a file at any given point in time. In order for several links to write to the same file, it needs to be a database table (you can use a hash file in Server). You can also write to severl distinct sequential files and, after the jobs have finished, concatenate these files together using DS.

[kcbland]

To add onto Arnds points, you also have an UNRESTARTABLE design, which you should always try to avoid.

Job A --> creates sequential file and writes 1000 rows.

Job B --> appends to file 1000 rows = 2000 rows total

Job C --> appends to file 1000 rows = 3000 rows total

Job D --> appends to file 500 rows and aborts = 3500 rows total

Job D --> restarts and appends to file 1000 rows = 4500 rows total

See the problem? Unless you build Job D to "skip" ahead already processed rows and pick up where it aborted, your sequential file is not what you needed. Now you either have to backup that file before each job writes to it, or take other measures. What if you want to RESTART Job D and have it REDO all rows, not just pick up where it left off. Now, your errorred rows may not error again because you fixed the problem.

Your best design is separate error files per job, and collate/concatenate as a later step. This is a BEST PRACTICE.

[babbu9]

HI
How would you concatenate flat files in datastage.

[kcbland]

Execute a Unix concatenate statement. If you carefully used a naming convention where there is a common portion of the filename, then simply use a cat statement with a wildcard.

Example: All of your jobs wrote out an error file for "FRED" table under the name "FRED_yourjobname.dat". You can then execute a "cat FRED_* > FRED.dat" statement. Now just pick your poison: write a Batch job and call the DSExecute API (search the forum); write a Sequence job and use a Routine stage to call ExecSH; call ExecSH in a before-job/transformer routine to concatenate the files when you need to access the combined file.

[Sainath.Srinivasan]

Best is to store the reject files in a separate directory with proper naming convention that will assist you in concatinating them together.

[babbu9]

Thank you for the info.