I think we are back to the same point from where we have started i.e. the processing of the 2 files means 2 separate jobs. Is this the final verdict in this issue or can I expect some other alternative.ray.wurlod wrote:Yes.
You can put the second link into a second job.
Execute the two jobs consecutively from a job sequence.
Sequential processing of Files
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 497
- Joined: Sun Dec 17, 2006 11:52 pm
- Location: Kolkata
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 497
- Joined: Sun Dec 17, 2006 11:52 pm
- Location: Kolkata
- Contact:
May be I was bit impolite in my reply, i apologise for that , can we think something on the lines of data set or file set instead of sequential files.ray.wurlod wrote:It's a free country - you can expect whatever you like.
But you won't get any alternatives from here, at least not from the experienced developers.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 497
- Joined: Sun Dec 17, 2006 11:52 pm
- Location: Kolkata
- Contact:
Can I modify my design using Copy stage as follows, The File(1) will go into the aggregator stage and then into the copy stage , from Copy stage we can derive 2 outputs , 1st output--file(2) and the 2nd output goes into the aggregator for the next level of aggregation. Though it may be bit cumbersome but it may reduce the number of jobs.ray.wurlod wrote:No. Your requirement (original post) was SEQUENTIALLY.
File Sets and Data Sets are read in parallel.
Please let me know whether this is feasible or not. I was trying to draw this process and paste but the format was not ok after pasting it as I am not aware about how to use the code.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
1. Click the Code button.
2. Draw your "ASCII art" picture.
3. Click the Code button again, to close the Code tags.
4. Click Preview.
5. Edit your "ASCII art" so that it looks better, then click Preview.
6. Repeat step 5 until the "ASCII art" looks like you want it to.
7. Click Submit.
Now think about partitioning and sorting requirements for the Aggregator stages.
2. Draw your "ASCII art" picture.
3. Click the Code button again, to close the Code tags.
4. Click Preview.
5. Edit your "ASCII art" so that it looks better, then click Preview.
6. Repeat step 5 until the "ASCII art" looks like you want it to.
7. Click Submit.
Code: Select all
-----> Aggreator 1 -----> Copy -----> Aggregator 2 ----> ????
|
+------> ????
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 497
- Joined: Sun Dec 17, 2006 11:52 pm
- Location: Kolkata
- Contact:
Thanx for the instructions .ray.wurlod wrote:1. Click the Code button.
2. Draw your "ASCII art" picture.
3. Click the Code button again, to close the Code tags.
4. Click Preview.
5. Edit your "ASCII art" so that it looks better, then click Preview.
6. Repeat step 5 until the "ASCII art" looks like you want it to.
7. Click Submit.
Now think about partitioning and sorting requirements for the Aggregator stages.Code: Select all
-----> Aggreator 1 -----> Copy -----> Aggregator 2 ----> ???? | +------> ????
I intend to modify my design by incorporating Copy stage in the job.
Code: Select all
File (2)
|
|
File(1)--->Aggregator---->Copy Stage---->Aggregator---->File(3)
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Anything that works is feasible.
It's not actually a bad design. But think about inserting some Sort stages ahead of each Aggregator stage - the second one can specify "don't sort (already sorted)" and thereby introduce some efficiency to the design.
It's not actually a bad design. But think about inserting some Sort stages ahead of each Aggregator stage - the second one can specify "don't sort (already sorted)" and thereby introduce some efficiency to the design.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 497
- Joined: Sun Dec 17, 2006 11:52 pm
- Location: Kolkata
- Contact:
then it could be something like:
File-----> Transformer-----> Transformer------> File
|-----> Agregator
File-----> Transformer-----> Transformer------> File
|-----> Agregator
pravin1581 wrote:The intermediate file is our output for one level of aggregation.Cr.Cezon wrote:Hi,
a possible solution could be :
File-----> Transformer-----> Transformer------> File
write an intermediate file has no sense in paralell
regards,
Cristina
File-----> Transformer1-----> Transformer2------> File
|-----> Agregator
transformer1 has two output links:
one to transformer2 and another to agregator.
|-----> Agregator
transformer1 has two output links:
one to transformer2 and another to agregator.
[/quote]pravin1581 wrote:The intermediate file is our output for one level of aggregation.Cr.Cezon wrote:Hi,
a possible solution could be :
File-----> Transformer-----> Transformer------> File
write an intermediate file has no sense in paralell
regards,
Cristina
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Intermediate files (as I mentioned in my initial response to this thread) are not permitted in parallel jobs. They are "blocking operations" and interfere with pipeline parallelism. If you need two streams, insert a Copy stage.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 497
- Joined: Sun Dec 17, 2006 11:52 pm
- Location: Kolkata
- Contact:
-
- Premium Member
- Posts: 497
- Joined: Sun Dec 17, 2006 11:52 pm
- Location: Kolkata
- Contact:
As per your suggestion I have incorporated sort stage to sort my data sequentially but the aggregator stage after that is rearranging the sorted data and mu output is getting unsorted.ray.wurlod wrote:Anything that works is feasible.
It's not actually a bad design. But think about inserting some Sort stages ahead of each Aggregator stage - the second one can specify "don't sort (already sorted)" and thereby introduce some efficiency to the design.