DSXchange

Hi ,

Is it possible to write to a same sequential file from two transformer stages in a job ? The metadata written by both the transformer stage is same.I tried doing this by giving overwriting existing file in one link & Append to existing file in other seq file link.

Cool....Its working fine....Just tested....be careful abt the the sequence...It shouldnt happen that the link you have set to overwrite the file gets started after the other one....

Hi,
writing to a sequential file in parallel is not supported and might cause unreliable data!
if you want simply to write to a sequential file twicw in the same job, again not in parallel, you'r better off writing 2 seperate stages each writing at it's time, this will probably reduce the chance of future changes to the job won't cause parallel writing ot htis file (I'd add an anotation specifying this in the job as well).

IHTH

I was experimenting now and found that it goes in a sequential mode. So I am wondering in what scenarios, the data can be unreliable.

I guess unless for constrainsts, which sequential files are stupid at....rest everything should go fine.

sdevashis wrote:I was experimenting now and found that it goes in a sequential mode. So I am wondering in what scenarios, the data can be unreliable.

I guess unless for constrainsts, which sequential files are stupid at....rest everything should go fine.

HEY HOW DID YOU DO THAT? I GOT AN ERROR WHEN I TRIED DOING THAT WITH FIXED WIDTH FILES.

Hi,
If some writing goes from both links at the same time to the file, data may be unreliable.
a sequential file stage is not ment to have parallel writing performed on it.
if you can be sure this won't happen, do it at your own risk.

I would prefer to seperate the 2, you could build 2 seperate files with the same definition and concatenate them when you'r done to form the final file you need (whethere be it via pure DS or an OS command run from DS)

IHTH

YOU CANNOT WRITE TO THE SAME FILE CONCURRENTLY IN EITHER UNIX OR NT. THIS IS NOT DATASTAGE.

Unix buffers writes to files in chunks of bytes, this is THE WAY IT WORKS. So if you have simultaneous processes trying to write to the same file, you will NOT interleave rows, you with interleave BYTES. Carriage returns and linefeeds are data, not ROW DELIMITERS to the operating system. You will end up with GARBAGE data, with some lines of clarity.

On NT you get into sharing violation issues. You also can get the same problem of as Unix of delayed writes.

If you must write to the same file from two transformers, the file cannot be a flat file. I have been successful with multiple sources writing to a hash file.

... tom

Here's an alternative solution:

Output to 2 separate files

After Job Routine upon successful run:

Merge the two files.

Simple.

-T.J.