Hi,
In my job, I am accessing a dataset for lookup purpose at five places. At all the Lookup, key columns are different. Dataset contains around 0.4 million records. Is it recommendable to access the same copy of dataset simultaneously while looking-up, as in my scenario?
Secondly, as lookup dataset is not quite small, I wanted to use Merge or Join instead of Lookup. But I want to capture all the rejected records while matching. So, left with Lookup option only. Join doesn't support Reject link, Merge rejects Update link's records not Master link's. Again, I had to go with Normal lookup as I cannot use Sparse lookup with dataset.
Any suggestions for better design?
Thanks in advance!
multiple access of a dataset within a job
Moderators: chulett, rschirm, roy
multiple access of a dataset within a job
Nitin Jain | India
If everything seems to be going well, you have obviously overlooked something.
If everything seems to be going well, you have obviously overlooked something.
On one hand, there shouldn't be any issue referencing the same dataset multiple times in the same job - that really isn't any different than multiple individual jobs accessing the same dataset concurrently and that works just fine.
On the other hand, the copy stage would allow you to read the dataset once and then have multiple copies in your process. The difficulty there is how to layout your job - you could end up with links going all over the place.
Brad.
On the other hand, the copy stage would allow you to read the dataset once and then have multiple copies in your process. The difficulty there is how to layout your job - you could end up with links going all over the place.
Brad.
I'm not sure if I fully understand your question, using multiple look-ups against one Dataset. I have used multiple look-ups for the same dataset using the Copy Stage. Each lookup is performed by using its own unique set of fields. This will enable you to capture your rejections and will centralize your lookup.
Charlie
I'm not sure it would be classed as good practice but until an enhancement is made that lets you put a bend in a link, you can achieve the same with a copy stage.
If the copy stage isn't doing anything but passing the columns straight through and Force = False it wil be optimised out when the job is compiled.
Of course if your job really is like a spiders web, rather than just a couple of links crossing over, it's probably better from a maintenance perspective to split the job into several smaller ones.
If the copy stage isn't doing anything but passing the columns straight through and Force = False it wil be optimised out when the job is compiled.
Of course if your job really is like a spiders web, rather than just a couple of links crossing over, it's probably better from a maintenance perspective to split the job into several smaller ones.