Merge 240 flat-files into one and then process using DataSta

[I_Server_Whale]

Hi Gurus,
I have 240 flat-files, each having the same number of fields (15). The files are located on a Windows Box (on a network). The DataStage 7.5.1 server and clients are on a windows box as well.

How do I merge these 240 files? I need to transform after merging these files. If I use a Merge Stage (Do we have merge stage in 7.5.1 Server job or is it called row-merger?), does it create any performance issues? If so, is there any other way of tackling this?. Any suggestion or advice is tremendously appreciated.

Thanks in Advance,
Naveen.

[chulett]

Use a script/bat before job to concatenate them all together, then process the concatenated file.

[ArndW]

I agree with Craig - even though DS can process files and file lists, all you need to do is a "type c:\mydir\*.txt >> outfile.txt" in dos and skip the lines that contain empty or just the filename. Or get the free unix commands for dos and use the cat command.

[I_Server_Whale]

Thank you Craig and ArndW. Say if I have three flat-files, ff1.txt, ff2.txt and ff3.txt in the "Merge" folder of the C drive.

Will the command look something like this?

Code: Select all

 C:\Merge\*.txt >> output.txt 

Where should I execute this command in windows?

Thanks again,
Naveen.

[Sainath.Srinivasan]

Merge cannot be run from DOS / Windows command prompt.

You may need to either write a routine or call from OS level API.

You need to code so that the job merges 2 files at a time and does for all the 240 files.

[ds_user78]

I believe you can do something like type c:\merge\*.txt > c:\merge\outfile.txt This you have to execute at the command prompt.

[ray.wurlod]

Try not to have the target file in the same folder as the source files.

Code: Select all

type c:\merge\*.txt > c:\temp\outfile.txt 

Or simply speccify type c:\merge\*.txt as the filter in a Sequential File stage.

[Sainath.Srinivasan]

Doh!! So you were referring to 'concatenate' the files and not DataStage 'merge'. I got confused.

As Ray suggested, you can either use any DOS / Win command for concatenating and then use it in the job or use the command directly in the filter stage.

[I_Server_Whale]

Thanks a lot!
Ray, ds_user78, ArndW, chulett and Sainath.

Naveen.

[shawn_ramsey]

Thanks a lot!
Ray, ds_user78, ArndW, chulett and Sainath.

Naveen.

I am not to sure if you found the actual command that you want. The correct syntax is:

Code: Select all

copy c:\merge\*.txt /a c:\temp\combofile.txt /a

[I_Server_Whale]

hello Shawn,
Yes! You are right. That is the correct syntax of the command. Thanks a lot,

Naveen.