DSXchange

Hi Gurus,
I have 240 flat-files, each having the same number of fields (15). The files are located on a Windows Box (on a network). The DataStage 7.5.1 server and clients are on a windows box as well.

How do I merge these 240 files? I need to transform after merging these files. If I use a Merge Stage (Do we have merge stage in 7.5.1 Server job or is it called row-merger?), does it create any performance issues? If so, is there any other way of tackling this?. Any suggestion or advice is tremendously appreciated.

Thanks in Advance,
Naveen.

Use a script/bat before job to concatenate them all together, then process the concatenated file.

I agree with Craig - even though DS can process files and file lists, all you need to do is a "type c:\mydir\*.txt >> outfile.txt" in dos and skip the lines that contain empty or just the filename. Or get the free unix commands for dos and use the cat command.

Thank you Craig and ArndW. Say if I have three flat-files, ff1.txt, ff2.txt and ff3.txt in the "Merge" folder of the C drive.

Will the command look something like this?
Where should I execute this command in windows?

Thanks again,
Naveen.

Merge cannot be run from DOS / Windows command prompt.

You may need to either write a routine or call from OS level API.

You need to code so that the job merges 2 files at a time and does for all the 240 files.

I believe you can do something like type c:\merge\*.txt > c:\merge\outfile.txt This you have to execute at the command prompt.

Try not to have the target file in the same folder as the source files. Or simply speccify type c:\merge\*.txt as the filter in a Sequential File stage.

Doh!! So you were referring to 'concatenate' the files and not DataStage 'merge'. I got confused.

As Ray suggested, you can either use any DOS / Win command for concatenating and then use it in the job or use the command directly in the filter stage.

Thanks a lot!
Ray, ds_user78, ArndW, chulett and Sainath.

Naveen.

naveendronavalli wrote:Thanks a lot!
Ray, ds_user78, ArndW, chulett and Sainath.

Naveen.

I am not to sure if you found the actual command that you want. The correct syntax is:

hello Shawn,
Yes! You are right. That is the correct syntax of the command. Thanks a lot,

Naveen.