row by row processing in datastage

[mahi_mahi]

Hi all,
I need to do row by row processing in datastage

example:
I have JOB1 and JOB2 , JOB1 is writing some 10 records in output file
FILE1.
for each record of FILE1 i need to process JOB2 and write output.

any suggestions please welcome

regards
mahi

[kcbland]

Custom job control. Look at a Batch job to get you started, all you need to do is capture the contents of FILE1 into an array and setup a loop. Consider something like this:

Code: Select all

UNIXcmd="cat ":FILE
Call DSExecute("UNIX", UNIXcmd, ScreenOutput, ReturnCode)

LineCount = DCOUNT(ScreenOutput, @AM)
For LinePtr=1 to LineCount
   Line=ScreenOutput<LinePtr>
   ...
   ...  Do whatever you need to do with this line
   ...  Your jobcontrol commands
   ...
Next LinePtr

[pkomalla]

I think u need to use loop condition while running those jobs using a shell script. In that, run the first job and then for each row of the output file run run the second job.

kumar

[DSguru2B]

I am really curious, mahi_mahi, can you please tell me why you want the data to be processed row by row. If you can tell us what is leading you to decide upon row by row processing then maybe we can come up with a better solution, performance wise and practice wise.

[mahi_mahi]

Thank you very much for your replies..My process is:

I will be receiving set of files daily
and i have set of conditions(abt 100) to check on each of these file on each of record in it...

PROCESS A
In this I will read the list of files file (a file which will have list of file names wht i recvd) record by record

for each record (it will have the filename to be processed) i need to run process B

PROCESS B
in this process it will apply all those 100 conditions
on each record and after applying conditions I will derive a flag which
tells me about discard the record or not

Now How can implement this process PROCESS A and B in datastage
i.e for each record process in A should run B

as pkomalla mentioned I can use shell script for process A and call Process B
but how can I halt the next triggering of Process B until the first processing completed

[DSguru2B]

As Ken mentioned, you need a custom job control. Go for Ken's solution. If your DS server was on unix, you could have done it via a script

[chulett]

If you have version 7.5.x then - Sequence job: Start Loop, End Loop and UserVariables Activity stages will get you what you need.

[ray.wurlod]

If you are on an earlier version than 7.5, you can call the UtilityRunJob routine from within a Transformer stage in the job that is processing the ten rows.

[kadf88]

If you have version 7.5.x then - Sequence job: Start Loop, End Loop and UserVariables Activity stages will get you what you need.

can the start/end loop stages read a uservariable activity variable?

[chulett]

Start loop, sure... as can downstream jobs. End loop doesn't need to.

[kadf88]

where can i define variables in the start loop stage?

[chulett]

Use can use User Variables in the Loop Definition 'Delimited Values' prompt.

[kadf88]

Chulett,

I have a flat file with say, 4 rows, how do i tell the loop to go through my sequence of jobs 4 times and pass elements from each row into my sequence?

thanks

[chulett]

Please start a new thread rather than (further) hi-jacking this one.

[kadf88]

Please start a new thread rather than (further) hi-jacking this one.

good point, I just thought it was relevent to the OP but I started a new one here:

viewtopic.php?t=100317