Reading multiple files in server jobs

[kirankumarreddydesireddy]

Hi

We had requirement to read from 6000 text files(.txt) and load them into single Oracle table.
We dont take any consideration of the format of the file(directly dump into oracle table).The output oracle table has only one column.

for example :

Input :

The test1.txt has 2 rows.

01*WOLAS*350-1*OBC-S*0%%N%12791%14387%726%570%698%636%473%719%607%855%732%841%739%632%853
01*ACCHK*700-1*OBC-A*0%%N%12793%14387%80%74%84%76%70%72%68%79%63%88%71%101%80

The test2.txt has 2 rows.

01*ACACRA*400-1*OBC-A*0%%N%12805%14367%19%16%16%17%18%17%21%18%17%18%22%20%15
01*OEINV*101-3*OBC-I*0%%N%12793%14387%142%124%150%154%118%206%246%251%186%186%154%158%178

Output :

The output oracle table will have 4 rows and one column.The entire value in the first row in the source file is written into first column of the table..

We are using 7.5,Server jobs in windows environment.Is there any optimized way we can do it, other than using multiple instance job and calling it multiple times which is very tedious?

Is there any stage which does this in Server jobs?

Thanks
Kiran

[chulett]

You need to treat them as one large file, either buy concatenating them all together first or by using the Filter option in the Sequential File stage and using 'cat/type *' to send the contents to std out, which the stage will then consume.

[kirankumarreddydesireddy]

Hi Chulett,

As I have mentioned earlier that we are using Windows and not UNIX(we cannot use cat function).

Is there any way to concatenate all of them into one large file using Datastage 7.5 Server job.

Will the folder stage does this?



Thanks
Kiran

[ray.wurlod]

TYPE is the DOS equivalent of the UNIX cat command. That's why Craig mentioned both.

[chulett]

Exactly. And no, the Folder stage doesn't do this. While it can read multiple files it is primarily meant to be used with XML files - which come in as one long string and are then parsed into individual fields in job.

[boggavarapu]

TYPE is the DOS equivalent of the UNIX cat command. That's why Craig mentioned both. ...

Hi Ray,

I am new to the Datastage. Can u please explain how to use TYPE in-order to
combine all the files data into single file.

[SURA]

You can use a copy command.

copy/b file1+file2 targetfile

[ray.wurlod]

Code: Select all

TYPE file1 file2 file3 ...

You can also use a cat command, even though you are on Windows. You have MKS Toolkit (a UNIX emulator) installed.

[SURA]

Yes Ray, that is the first ans came in my mind and i was surprised why it wan not mentioned before....

[chulett]

Yes Ray, that is the first ans came in my mind and i was surprised why it was not mentioned before....

You mean other than in the very first response?

[SURA]

YES: Before i response i got this in my mind;

[ray.wurlod]

That's really a different question and should be a new topic.

[chulett]

Not really a question at all but rather another proposed solution...