How to start a Sequencer from Mainframe

[shamshad]

Our ETL load 70 tables in our warehouse. All the sources are text file (.TXT).

A mainframe job ftps all the source file to the Data Stage server. What we want to achieve is when the LAST SOURCE FILE is COPIED successfully to the Data Stage server, we want to trigger the MASTER JOB in Data Stage.

How can this be done? Please share your views. Any help will he greatly appreciated.

[lstsaur]

The way I have done before is that after your mainframe has FTP'ed all the files, it also sends out a flag to the specific directory of the DS server. Then your sequnece job has a Wait_For_File_Activity first to pick up this flag and subsequently to process other jobs in the sequence.
Hope this will give you an idea.

[kcbland]

If you absolutely wish the mainframe processes to initiate the load, and avoid a scheduler, then you simply need to start the jobstream using the command line program dsjob on the unix server. You can invoke this using any means your mainframe has for running remote programs on other machines. In unix we'd suggest ssh, remsh, rsh, rlogin, etc.

[ray.wurlod]

If the FTP transfers are under control of an enterprise scheduler, such as Control-M, SeeBeyond, etc., then the same scheduler can be used to initiate the DataStage job sequence.

[avi21st]

Our ETL load 70 tables in our warehouse. All the sources are text file (.TXT).

A mainframe job ftps all the source file to the Data Stage server. What we want to achieve is when the LAST SOURCE FILE is COPIED successfully to the Data Stage server, we want to trigger the MASTER JOB in Data Stage.

How can this be done? Please share your views. Any help will he greatly appreciated.

Hi

We have done somthing like that -

One way is ---

Code: Select all

you can pass the Unix shell script name in the PARAM part of your JCL
Now once the JCL triggers the process it triggers the datastage job from unix by a dsjob -run

Another way which I have implemented is by file watcher concept using two unix daemon scripts.

Code: Select all

A Unix daemon checks for new files say in a Inbox directory. It compares file size to check if transfer is done. then move the file to SrcFiles folder with timestamp appended. Then a daemon process running in Unix wakes up in a certain interval- gets all the file available. I use a FILE_LIST table and have a column called process_flg. So the second daemon checks the file with  process_flg as Y and kick of that job via dsjob run

These are very standard process-let me know if you have questions

[venkat_kp]

Avishek Mukherjee,

Appreciate if you can share the unix code in a generic way.


Thanks.

Another way which I have implemented is by file watcher concept using two unix daemon scripts.

Code: Select all

A Unix daemon checks for new files say in a Inbox directory. It compares file size to check if transfer is done. then move the file to SrcFiles folder with timestamp appended. Then a daemon process running in Unix wakes up in a certain interval- gets all the file available. I use a FILE_LIST table and have a column called process_flg. So the second daemon checks the file with  process_flg as Y and kick of that job via dsjob run

These are very standard process-let me know if you have questions