How to wait until all the data on a link is processed?
Moderators: chulett, rschirm, roy
How to wait until all the data on a link is processed?
Hi,
I have a job which reads data from a file. The file has data in the following format
Header
Detail
Detail
Detail
Detail
Trailer
The job design is simple. I have a sequential file stage followed by a transformer which separates the header, detail and trailer.
The header link will go and insert into a table and likewise for the detail
Once the trailer message comes it will kick off a function that will update the data inserted by the detail flow.
The problem is that, before the last detail message is inserted, the trailer is fetched and the function is not updating the last detail message.
I want to know if there is any way for the job to wait until the detail messages are processed and then fetch the trailer?
I have a job which reads data from a file. The file has data in the following format
Header
Detail
Detail
Detail
Detail
Trailer
The job design is simple. I have a sequential file stage followed by a transformer which separates the header, detail and trailer.
The header link will go and insert into a table and likewise for the detail
Once the trailer message comes it will kick off a function that will update the data inserted by the detail flow.
The problem is that, before the last detail message is inserted, the trailer is fetched and the function is not updating the last detail message.
I want to know if there is any way for the job to wait until the detail messages are processed and then fetch the trailer?
What target stage are you using?
You could try putting a reject link from the detail insert stage and placing that as a referential lookup into the msg trailer link. This should, in theory, make the flow from the msg trailer link wait until all the inserts into the table have finished, then continue on from the lookup to use the msg trailer link to do what ever it has to do.
Once again only a theory, not sure if it will actually work.
Or you could just do it in two different jobs.
You could try putting a reject link from the detail insert stage and placing that as a referential lookup into the msg trailer link. This should, in theory, make the flow from the msg trailer link wait until all the inserts into the table have finished, then continue on from the lookup to use the msg trailer link to do what ever it has to do.
Once again only a theory, not sure if it will actually work.
Or you could just do it in two different jobs.
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
Yes Sainath, the job is running on a single node.Sainath.Srinivasan wrote:Not fully sure what you need.
Did you try running that stage(s) / job in a sequential mode or in 1 node config ? ...
And my problem is even before my detail record is insert into the database my trailer record is calling a function which will updated a column on the inserted details. Effectively missing out on updating the last detail column. I just want my trailer to wait till all the detail message are laoded
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
I take it this means that your custom stage takes all the input streams for each file segment (header, detail, trailer) then updates using the function within the stage?yabhinav wrote: ... we are using a custom stage called UOW.
So i will not be able to have reject links on it.
I can't use two different jobs because I need to pass some values coming in from the trailer to the function.
If that is the case, we can't really comment without knowing what the stage actually does.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Sorry, I was away for a while.ray.wurlod wrote:Care to share?
...
We read Mq messages from a custom stage, so we added extra parameters to the stage to call a C function.
This C function will check if the incoming record is a trailer or not.
If it is, it will hold the trailer till all the messages in the queue are flushed i.e all the messages are processed and EOW has been received for each of them. After this the trailer will be processed