Merging Files

[jo8712]

I have two Sequential files that with all the same colums but different data and I want to merge them into one sequential file. How can I do this is DataStage. The Merge Stage looks to be the obvious choice at first but it does not take streaming data inputs from a Seq file. Any ideas.

[JDionne]

I have two Sequential files that with all the same colums but different data and I want to merge them into one sequential file. How can I do this is DataStage. The Merge Stage looks to be the obvious choice at first but it does not take streaming data inputs from a Seq file. Any ideas.

They way that I did it in DS (and this isnt the most efective way in terms of performance) is that I used two link partitioners into one ling collector. that way I was also able to do some data cleansing through transformer stages and it went into one file as an output of the collector. It has been raised to my attention that this method isnt the best way to optimize the process and it has been recommended to me to use a copy command to copy the files into one file ie copy file1 + file2 file3 I am actualy still trying to see if this will work. I have goten an error when I try to use the output of that copy command. I am looking into if that was a data problem or what. Eather way will do the trick. BTW if u do the copy method then u will need to write a batch file and execute it after job completion.

Lets overview cause i have confused myself.
if u use the copy command run your sequencial files through transformers into other sequencial files and then exec a dos batch file (if in windows) to copy them together.

or you can use two link partioners into one link collector

Jim

[raju_chvr]

As you are on UNIX system, why don't you use the 'cp' command of UNIX and make the 2 files into 1(as u said they have same MetaData) and then do the Transformations on the file you have just created. You can run this 'cp' command as a before-job sub-routine from job properties.

Well it worked for me and is working for me.

[kcbland]

The command syntax would be:

Code: Select all

cat file1 file2 file3 > combinedfile

Better yet, a wildcard will get all matching files:

Code: Select all

cat file* > combinedfile

Do this in a command stage or via a BATCH job DSExecute call prior to the job needing it. Modularity folks, keep that in mind.

[roy]

Hi,
in case you only need to concatenate the files they are all correct.
the merge stage actually lets you merge 2 files as if they were 2 tables and use one of the join methods to effect the output you'll get.
so if you wanted to get a single line with same key columns from the 2 files the merge stage would be the way.
BTW the merge stage can only change the output derivation that means columns from eighther file, but not make a new column with lets say file1.col+file2.col as it's value.
also the merge stage must output all key columns that comprise the join criteria between the files(filter if needed in a following transformer).

IHTH

[Teej]

I have two Sequential files that with all the same colums but different data and I want to merge them into one sequential file. How can I do this is DataStage. The Merge Stage looks to be the obvious choice at first but it does not take streaming data inputs from a Seq file. Any ideas.

If you must use DataStage, use the Funnel stage.

-T.J.