writing to a same sequential file from two transformer stage

raj_cipher · Post by **raj_cipher** » Mon Jan 12, 2004 1:18 am

Hi ,

Is it possible to write to a same sequential file from two transformer stages in a job ? The metadata written by both the transformer stage is same.I tried doing this by giving overwriting existing file in one link & Append to existing file in other seq file link.

sdevashis · Post by **sdevashis** » Mon Jan 12, 2004 2:26 am

Cool....Its working fine....Just tested....be careful abt the the sequence...It shouldnt happen that the link you have set to overwrite the file gets started after the other one....

roy · Post by **roy** » Mon Jan 12, 2004 2:36 am

Hi,
writing to a sequential file in parallel is not supported and might cause unreliable data!
if you want simply to write to a sequential file twicw in the same job, again not in parallel, you'r better off writing 2 seperate stages each writing at it's time, this will probably reduce the chance of future changes to the job won't cause parallel writing ot htis file (I'd add an anotation specifying this in the job as well).

IHTH

sdevashis · Post by **sdevashis** » Mon Jan 12, 2004 2:39 am

I was experimenting now and found that it goes in a sequential mode. So I am wondering in what scenarios, the data can be unreliable.

I guess unless for constrainsts, which sequential files are stupid at....rest everything should go fine.

raj_cipher · Post by **raj_cipher** » Mon Jan 12, 2004 4:49 am

sdevashis wrote:I was experimenting now and found that it goes in a sequential mode. So I am wondering in what scenarios, the data can be unreliable.

I guess unless for constrainsts, which sequential files are stupid at....rest everything should go fine.

HEY HOW DID YOU DO THAT? I GOT AN ERROR WHEN I TRIED DOING THAT WITH FIXED WIDTH FILES.

roy · Post by **roy** » Mon Jan 12, 2004 5:38 am

Hi,
If some writing goes from both links at the same time to the file, data may be unreliable.
a sequential file stage is not ment to have parallel writing performed on it.
if you can be sure this won't happen, do it at your own risk.

I would prefer to seperate the 2, you could build 2 seperate files with the same definition and concatenate them when you'r done to form the final file you need (whethere be it via pure DS or an OS command run from DS)

IHTH

kcbland · Post by **kcbland** » Mon Jan 12, 2004 6:47 am

YOU CANNOT WRITE TO THE SAME FILE CONCURRENTLY IN EITHER UNIX OR NT. THIS IS NOT DATASTAGE.

Unix buffers writes to files in chunks of bytes, this is THE WAY IT WORKS. So if you have simultaneous processes trying to write to the same file, you will NOT interleave rows, you with interleave BYTES. Carriage returns and linefeeds are data, not ROW DELIMITERS to the operating system. You will end up with GARBAGE data, with some lines of clarity.

On NT you get into sharing violation issues. You also can get the same problem of as Unix of delayed writes.

tomengers · Post by **tomengers** » Mon Jan 12, 2004 12:15 pm

If you must write to the same file from two transformers, the file cannot be a flat file. I have been successful with multiple sources writing to a hash file.

... tom

Teej · Post by **Teej** » Mon Jan 12, 2004 4:22 pm

Here's an alternative solution:

Output to 2 separate files

After Job Routine upon successful run:

Merge the two files.

Simple.

-T.J.