Processing files dynamically

satishkt · Post by **satishkt** » Tue Jul 05, 2005 11:33 am

I have a rather unusual requirement with DataStage. There is a requirement where we have buiness users submitting files(csv) all the time and DataStage should be able to process that file the instant it is submitted. Basically DS extracts the file to db after which the user can trigger off a rule engine which acts on the data that was just submitted and then a report is generated.

The initial thought process was that we would have an upload page in the webapp which would copy the file to a specified location and somehow (i still not have figured how) get DataStage to process that file. But because of certain other constraints that went away, the final deal is that users are going to paste that file on a windows share(thru explorer) on their desktop (the share is mounted on a unix box actually). Once the file is placed in a pre designated folder , DataStage has to pick up the file and process it. And since users keep working on the file all the time , they can copy the file onto the pre designated location as many times as they want and DS should be able to process it each time they do it.

I understand that they are potentially multiple problems with this approach , but given the constraints this is the approach we have to take. What I am looking for is ideas/best possible approaches around how to solve this problem with an eye on the maintability of the solution (error handling/reporting).

I am on DataStage Server Edition v 7.5 on Solaris. Alternatively I heard there is an alternative approach where i can invoke DS via java as a webservice or something. Also that on of the DS versions enables java integration do let me know where i can find more info on them coz if i can invoke DS from my java web app, i would probably able to go with the approach of a file upload.

thanks
-Satish

tonystark622 · Post by **tonystark622** » Tue Jul 05, 2005 12:17 pm

Can you access the files on the windows share from DataStage? If so, one way to do this would be to set up a shell script that periodically ftp's to the windows share and gets the file (if it's there). Then invokes DataStage on Unix with the 'dsjob' command.

Tony

ukyrvd · Post by **ukyrvd** » Tue Jul 05, 2005 12:28 pm

Is DSWaitForFile before/after sub-routine an option?? This basically waits on a file in a specified location and as soon it finds the file starts processing. Again this depends upon the accessibility of the windows share from DataStage

pnchowdary · Post by **pnchowdary** » Tue Jul 05, 2005 12:30 pm

Hi Satish,

I believe you can use Wait-For-File-Activity stage in the job sequencer and wait for the file to appear in the designated directory. After the file appears, you can kick of a datastage job, which processes the input file according to your transformation rules.

Thanks,
Naveen

ray.wurlod · Post by **ray.wurlod** » Tue Jul 05, 2005 3:22 pm

Yes, you can expose your DataStage job as a web service. Search the forum for RTI (real time interface). This is probably a good approach in this case.