create and append text to sequential file in the same stage

chanthou_2000 · Post by **chanthou_2000** » Fri Apr 15, 2005 6:54 am

Hello everyone,

i'm not an beginner in designing DS job, but i tried to create and append text file, but it seems that the "append" action overwrote the current first rows (from top to bottom). It's very annoying. I have to precise that there is only one source (text file), one transformer, and from this transformer, there are two seq. file stage, the first to create the file, and the second to append "transformed" rows and i set the correct link order (1- create 2-append). But unfortunately, the "append" overwrote from the top and not from the bottom of the text file. What is wrong ? Is it a bug ? i was thinking that the "append" action always write from the end of the file.

Thanks for your answers

Chanthou

chulett · Post by **chulett** » Fri Apr 15, 2005 7:05 am

This seems to be a popular topic lately.

For one example, please see this thread in the EE/PX forum.

You cannot have two processes writing to the same sequential file at the same time. Period. If your create link is writing rows to the file, it must complete and close the file before it is safe for the append action to begin. It's not a bug or a DataStage problem, it is how sequential files work.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Fri Apr 15, 2005 7:06 am

Write in 2 separate seq files and concat them at end.

wilcywilliam · Post by **wilcywilliam** » Fri Apr 15, 2005 7:09 am

The output to the sequential file is in the same order as the input. So if the rows first satisfy the 'append' constraint, then those rows are written first. Specifying the link order in sequential file stages doesn't ensure that all outputs to the 'create' link are written first followed by the 'append' link.

chulett · Post by **chulett** » Fri Apr 15, 2005 7:11 am

Sainath.Srinivasan wrote:Write in 2 separate seq files and concat them at end.

Exactly. Or do that and then concatenate in the 'after job' area via ExecSH.

chanthou_2000 · Post by **chanthou_2000** » Fri Apr 15, 2005 7:35 am

Thanks for your answer, but finally i decided to create 2 separated jobs, one to create and one to append.

Thanks again

ray.wurlod · Post by **ray.wurlod** » Fri Apr 15, 2005 4:40 pm

That's what you had to do.

The "create" stage has not closed the file, and chances are that it had not actually created the file (rows had been buffered in memory) before the "append" stage was activated and therefore determined that it, too, must write to a new, empty file.

With sequential files, there can only be one writer at a time, so the updates through the earlier stage were overwritten.

Your separated solution is the correct one.