Is there any issues that need too be considered while using the Collector Stage in Datastage Server Jobs?????
I need to do a following thing in Server Job
Problem : If a value for a Particular Column is A.
then i need to do a lookup with a hash file. If it is not found or if the say COLUMN <> A,
then i need to do a lookup once again with another Hash file B , then Combine the Flows together and do further Processing.
For the Above Problem
I have used following Logic to Build
..........................HASH 1
SplitTrans -------> Trans1--------->Found
|...................................................|
|<-----------------NotFound.............Collector2----->FurtherTrans
Collector1.......................................|
|-------------------->Trans2------->Found
............................HASH2
Solution Used :
1) SPLIT TRANS for COLUMN =A and COLUMN <> A.
2) Then those records which failed lookup with Trans1 and Column<> a from Split trans (Used Collector1 stage to collect) and then go for a lookup with HASH2.....
3) Used Collector2 for succesful lookups of TRans1 and Trans2 Succesful lookups to again collect in Collector to send to FurtherTrans for further Transformations....
Phew!!!!!!!!! .
I guess ppl shld have understood the Logic.
So I am doing the Above stuff with Link Collectors.
Does it lead to any Performance Issues ????????
Or is there any Better way of Doing the Above Problem...
Link Collector Issue
Moderators: chulett, rschirm, roy
You can do like the following.
---------------------Hashlook for A
|
Source -----------Constraint Col=A--------------Output_A
---------------------Hashlook for <> A
|
Source -----------Constraint Col<>A--------------Output_Others
Then join those file together using your OS command (cat or copy)
Thanks
Siva
---------------------Hashlook for A
|
Source -----------Constraint Col=A--------------Output_A
---------------------Hashlook for <> A
|
Source -----------Constraint Col<>A--------------Output_Others
Then join those file together using your OS command (cat or copy)
Thanks
Siva
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Oops Right Ray.
But I hope u got my problem.
Concerns for your soln. Rasi is that :
1) Putting 2 queries on Source side will loads the Source because of which I don't have the Liberty i.e why I have a SplitTrans (Transformer) to do Splitting.
2) Also in the Solution HASH1 and HASH2 are two different hash files. those which fail lookup with HASH1 needs to be done a lookup again with hash2 i.e why I had a collector to do that.
So Is it not a good Practice to use a Collector to first Collect then Do a lookup ?????
Or using the hash file twice put two Trans (once for a lookup colum<>A)
(second for Column = A andd HASH1 lookup Failure)
and do the lookups.
Coming back my question still remains is using Link Collectors a Problem???
But I hope u got my problem.
Concerns for your soln. Rasi is that :
1) Putting 2 queries on Source side will loads the Source because of which I don't have the Liberty i.e why I have a SplitTrans (Transformer) to do Splitting.
2) Also in the Solution HASH1 and HASH2 are two different hash files. those which fail lookup with HASH1 needs to be done a lookup again with hash2 i.e why I had a collector to do that.
So Is it not a good Practice to use a Collector to first Collect then Do a lookup ?????
Or using the hash file twice put two Trans (once for a lookup colum<>A)
(second for Column = A andd HASH1 lookup Failure)
and do the lookups.
Coming back my question still remains is using Link Collectors a Problem???