Append into the LookUp Fileset?
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 57
- Joined: Wed Nov 15, 2006 10:25 am
- Location: Ottawa ON
Append into the LookUp Fileset?
Is it possible to append into the LookUp Fileset in a later job after it was created and loaded by a job?
Anyone knows?
Thanks in advance!
Anyone knows?
Thanks in advance!
TPQ
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
I don't think it's possible. There is no overwrite/append property in the Lookup File Set stage. But you can construct something like the following.
Code: Select all
LookupFileSetprevioussource
|
V
Funnel ----> LookupFileSet
^
|
(new data)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 57
- Joined: Wed Nov 15, 2006 10:25 am
- Location: Ottawa ON
Thanks for your response, Ray.
I tried but the LookupFileSetprevioussource doesn't support an output link into Funnel or anything. I did the test to put the data into a dataset at the sametime as into the LookupFileSetprevioussource. Later, I could append into the dataset in a different job and reload the LookupFileSet.
I'm just wondering about this kind of performance. Is there any better way?
Thanks a bunch!
I tried but the LookupFileSetprevioussource doesn't support an output link into Funnel or anything. I did the test to put the data into a dataset at the sametime as into the LookupFileSetprevioussource. Later, I could append into the dataset in a different job and reload the LookupFileSet.
I'm just wondering about this kind of performance. Is there any better way?
Thanks a bunch!
TPQ
-
- Premium Member
- Posts: 57
- Joined: Wed Nov 15, 2006 10:25 am
- Location: Ottawa ON
-
- Premium Member
- Posts: 1044
- Joined: Wed Sep 29, 2004 3:30 am
- Location: Nottingham, UK
- Contact:
That's right, it doesn't. Once your data goes into a LookUpFileset, the only thing you can do is to look up against it. You have to jump through hoops in order to accumulate data from several sources over time, re-creating the file set from another copy of the source data that can be streamed.Thanh Dalton wrote:When I use LookUpFileset as a source and Funnel stage as a destination and I got a message saying :The source stage does not support stream output links!
I don't think that Lookup File Sets are a good idea anyway. Your stream data has to be partitioned the same way that the Lookup File Set was partitioned when it was created, which is not efficient (you might have a small set of look up data, but have to re-partition millions of rows of stream data just to do the look up on that key). I put all my look up data in regular File Sets.
Phil Hibbs | Capgemini
Technical Consultant
Technical Consultant
-
- Premium Member
- Posts: 57
- Joined: Wed Nov 15, 2006 10:25 am
- Location: Ottawa ON
Thanks to Phil and ArndW!
My experience with the table to do the LookUp for 300M rows take so long. That's why I prefer to use LookUpFileset instead. It has proved to perform very well. In this case, I can append into a dataset and reload to the LookUp Fileset. I tested for 100M rows to load the LookUp Fileset from a dataset and it took me only 9 minutes. So I guess, I go for the dataset then!
Thanks to all for the help!
My experience with the table to do the LookUp for 300M rows take so long. That's why I prefer to use LookUpFileset instead. It has proved to perform very well. In this case, I can append into a dataset and reload to the LookUp Fileset. I tested for 100M rows to load the LookUp Fileset from a dataset and it took me only 9 minutes. So I guess, I go for the dataset then!
Thanks to all for the help!
TPQ
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: