Merging files

[anu123]

I have 12 source files as

file_1,file_2,....file_12.

I want merge them all to create a single file say 'file'

Is there any such stage in DataStage where we can give all input file names and get the merged file as output.

Basically I need 'cat file_1,file_2 .....file_12 > file'

[DSguru2B]

Use the cat command in the Filter options in the sequential file stage properties. Funnel Stage can get you what you need too.

[anu123]

Use the cat command in the Filter options in the sequential file stage properties. Funnel Stage can get you what you need too.

thanks guru.

If I go for 'funnel' I guess, my job looks ugly with '12' source stages.
I could not understand the 1st option. If I am not wrong, in sequential stage properties..


Options --> Filter = cat #path#file_1,#path#_file_2.... > #path#_file

thne what would be the

Source --> File = ?

thank you

[DSguru2B]

Good question. I am not sure what you are going to put there. Do this
In the filter command, get rid of the redirection. Basically your filter command will look something like this

Code: Select all

cat #path#file_1 #path#_file_2 cat #path#_file_3 ......#path#_file_12

In the File section put #path#file_1.
I think it will get the output from the filter command and neglect the File name. I tried a test and it works.
NOTE: Even though it seems its neglecting the file name, but it has to be a valid file name. Thats why I asked you to put #path#file_1 as it will already exist for the filter command.

[ray.wurlod]

Even easier (and better performing, because you will get parallel operation) would be to use multiple File properties in the Sequential File stage, or to change the read mode to "File Pattern" and specify a regular expression to select the files.

[anu123]

Even easier (and better performing, because you will get parallel operation) would be to use multiple File properties in the Sequential File stage, or to change the read mode to "File Pattern" and specify a regular expression to select the files.

thanks Ray.

I am unable to locate a 'Multiple File properties' option in sequential file stage. trying with 2nd option you gave me

I changed 'Read mode' to 'File Pattern' and in the 'File Pattern' files as
#path#file_1.csv #path#file_2.csv (used 2 files to test).

it thows an error as

Input buffer overrun at field "col2"


I know these silly questions but I ma pretty new to PX.thanks in advance.

[anu123]

Even easier (and better performing, because you will get parallel operation) would be to use multiple File properties in the Sequential File stage, or to change the read mode to "File Pattern" and specify a regular expression to select the files.

thanks Ray.

I am unable to locate a 'Multiple File properties' option in sequential file stage. trying with 2nd option you gave me

I changed 'Read mode' to 'File Pattern' and in the 'File Pattern' files as
#path#file_1.csv #path#file_2.csv (used 2 files to test).

it thows an error as

Input buffer overrun at field "col2"


I know these silly questions but I ma pretty new to PX.thanks in advance.

I figured it out. Its data type issue. Thanks all for you valuable time.

Ray would you mind to elaborate a bit more on the 'Multiple File properties' approach...

thanks a lot.

[DSguru2B]

Read in the sequential file stage help pdf file about "File Pattern" in "Read Method". I would say its a better way to handle your requirement. It also takes in wild cards and processes all the files.

[anu123]

Read in the sequential file stage help pdf file about "File Pattern" in "Read Method". I would say its a better way to handle your requirement. It also takes in wild cards and processes all the files.

thank you.

[ray.wurlod]

Read Method is "Specific File".
You can add more than one "File" property.

This is what I meant by multiple File properties.