Merging Files
Moderators: chulett, rschirm, roy
Merging Files
I have two Sequential files that with all the same colums but different data and I want to merge them into one sequential file. How can I do this is DataStage. The Merge Stage looks to be the obvious choice at first but it does not take streaming data inputs from a Seq file. Any ideas.
Re: Merging Files
jo8712 wrote:I have two Sequential files that with all the same colums but different data and I want to merge them into one sequential file. How can I do this is DataStage. The Merge Stage looks to be the obvious choice at first but it does not take streaming data inputs from a Seq file. Any ideas.
They way that I did it in DS (and this isnt the most efective way in terms of performance) is that I used two link partitioners into one ling collector. that way I was also able to do some data cleansing through transformer stages and it went into one file as an output of the collector. It has been raised to my attention that this method isnt the best way to optimize the process and it has been recommended to me to use a copy command to copy the files into one file ie copy file1 + file2 file3 I am actualy still trying to see if this will work. I have goten an error when I try to use the output of that copy command. I am looking into if that was a data problem or what. Eather way will do the trick. BTW if u do the copy method then u will need to write a batch file and execute it after job completion.
Lets overview cause i have confused myself.
if u use the copy command run your sequencial files through transformers into other sequencial files and then exec a dos batch file (if in windows) to copy them together.
or you can use two link partioners into one link collector
Jim
Sure I need help....But who dosent?
Re: Merging Files
As you are on UNIX system, why don't you use the 'cp' command of UNIX and make the 2 files into 1(as u said they have same MetaData) and then do the Transformations on the file you have just created. You can run this 'cp' command as a before-job sub-routine from job properties.
Well it worked for me and is working for me.
Well it worked for me and is working for me.
The command syntax would be:
Better yet, a wildcard will get all matching files:
Do this in a command stage or via a BATCH job DSExecute call prior to the job needing it. Modularity folks, keep that in mind.
Code: Select all
cat file1 file2 file3 > combinedfile
Code: Select all
cat file* > combinedfile
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Hi,
in case you only need to concatenate the files they are all correct.
the merge stage actually lets you merge 2 files as if they were 2 tables and use one of the join methods to effect the output you'll get.
so if you wanted to get a single line with same key columns from the 2 files the merge stage would be the way.
BTW the merge stage can only change the output derivation that means columns from eighther file, but not make a new column with lets say file1.col+file2.col as it's value.
also the merge stage must output all key columns that comprise the join criteria between the files(filter if needed in a following transformer).
IHTH
in case you only need to concatenate the files they are all correct.
the merge stage actually lets you merge 2 files as if they were 2 tables and use one of the join methods to effect the output you'll get.
so if you wanted to get a single line with same key columns from the 2 files the merge stage would be the way.
BTW the merge stage can only change the output derivation that means columns from eighther file, but not make a new column with lets say file1.col+file2.col as it's value.
also the merge stage must output all key columns that comprise the join criteria between the files(filter if needed in a following transformer).
IHTH
Roy R.
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Re: Merging Files
If you must use DataStage, use the Funnel stage.jo8712 wrote:I have two Sequential files that with all the same colums but different data and I want to merge them into one sequential file. How can I do this is DataStage. The Merge Stage looks to be the obvious choice at first but it does not take streaming data inputs from a Seq file. Any ideas.
-T.J.
Developer of DataStage Parallel Engine (Orchestrate).