Read and write from single dataset in job
Moderators: chulett, rschirm, roy
Read and write from single dataset in job
I'm trying to use a single dataset(once as lookup and again as target) in the same job. Can someone please let me know if this is possible as I keep getting the following error.
Operator initialization: A link between two operators should be named with a .v; insert a copy operator to save a persistent copy of the data
Thanks
Operator initialization: A link between two operators should be named with a .v; insert a copy operator to save a persistent copy of the data
Thanks
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Ray,
I guess the lookup file set doesn't suffer from the same restrictions as that seemed to work, although I'm still testing to establish whether the lookup fileset does the equivalent of a truncate and load rather than an append.
What in yor mind would be a reasonable approach to replicating the Server hash file lookup/write.
I guess the lookup file set doesn't suffer from the same restrictions as that seemed to work, although I'm still testing to establish whether the lookup fileset does the equivalent of a truncate and load rather than an append.
What in yor mind would be a reasonable approach to replicating the Server hash file lookup/write.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
There isn't one. Stop thinking like a server job developer. Start envisaging virtual Data Sets, and learn how an index to a virtual Data Set is built "on the fly" on a reference input link to a Lookup stage, except when that link is serviced by a Lookup File Set or when sparse lookup is specified.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
This is not a case for server job or enterprise job. It's problem how enterprise edition can handle it. Such as for surrogate key case: you generate a surrogate key for a new customer. In the same run, the new customer can be present in more than one service providers, so can have more than one instance. When you generate a key for the first instance, and when the second comes in, you need to find his/her key in the dataset, or it will generate another key for him/her.
So how do you handle this scenario?
There are other situations fit in the scenario, such as in web service, a session token from previous request, is used in following request, ...
I try looking up directly from a transfomer, Datastage states it's not allowd a cycle operation. I try it with sequential file, it's allowed, but look up returns nothing even a match is there.
Thanks,
So how do you handle this scenario?
There are other situations fit in the scenario, such as in web service, a session token from previous request, is used in following request, ...
I try looking up directly from a transfomer, Datastage states it's not allowd a cycle operation. I try it with sequential file, it's allowed, but look up returns nothing even a match is there.
Thanks,
If you are doing a normal lookup, then it reads all the reference data into memory before processing the main input data.olgc wrote:I try looking up directly from a transfomer, Datastage states it's not allowd a cycle operation. I try it with sequential file, it's allowed, but look up returns nothing even a match is there.
If you are able to do a sparse lookup, then it does not read any reference data into memory, rather it looks up each record one by one. Sparse lookup should handle your scenarios.
Choose a job you love, and you will never have to work a day in your life. - Confucius
Thanks, Eric, for your good suggestion. But only database table look up can be configured with sparse look up as statement below. For look up into sequential file, there is no sparse look up possible. Thanks,
Configuring sparse lookup operations
Data that is read by a database stage can serve as reference data to a Lookup stage. By default, this reference data is loaded into memory like any other reference link. When directly connected as the reference link to a Lookup stage, you can configure the Lookup Type property of the DB2 connector to Sparse and send individual SQL statements to the database for each incoming Lookup row.
I guess I don't understand what you are asking.
If both of them are needed in the system, then shouldn't each of them have unique keys generated?olgc wrote:In the same run, the new customer can be present in more than one service providers, so can have more than one instance. When you generate a key for the first instance, and when the second comes in, you need to find his/her key in the dataset, or it will generate another key for him/her.
So how do you handle this scenario?
Choose a job you love, and you will never have to work a day in your life. - Confucius
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI