Updating Techniques - Lookup Data [Reference]

vijayrc · Post by **vijayrc** » Mon Sep 11, 2006 5:34 pm

Hi,
I would like to know, how any static data file used in reference be updated in DataStage? Does DS have any mechanism to update, or any third party tool to be used for such updates. Any directions on this, is appreciated. Thanks in advance, Vijay

kumar_s · Post by **kumar_s** » Mon Sep 11, 2006 6:38 pm

What is your lookup data stage. You need to have a seperate flow in datastage to update the information of the static file. If the inputs are from third party, or web oriented, you need to capture it and place it in a common folder with a predefined filename, either in flatfile or in any database. So that datastage can read and update the the lookup file.
If you need to update the lookup file based on the input, it has its own flow.

ray.wurlod · Post by **ray.wurlod** » Mon Sep 11, 2006 9:38 pm

There's a logical inconsistency in your question. If it's static then surely there's no need to update it? Am I missing something here?

vijayrc · Post by **vijayrc** » Tue Sep 12, 2006 4:15 pm

ray.wurlod wrote:There's a logical inconsistency in your question. If it's static then surely there's no need to update it? Am I missing something here? ...

Sorry if I had not been clear. Say I have a reference file, which is going to be static e.g., Item# 222222 Description: Pencil which next month could be PEN. When such a change is needed, should the static reference file be updated using a separate adhoc flow, or is there any DS option/3rd Party tool available to update the reference file. Hope I made it clear.

ray.wurlod · Post by **ray.wurlod** » Tue Sep 12, 2006 4:21 pm

Parallel jobs are not equipped to do this kind of real-time updating without running in sequential mode and using auto-commit, both of which slow things down. It is regarded as a "blocking" activity. Also, you will not be able to get it to work with virtual Data Sets, so you're pretty much constrained to sparse lookups.

If the volume of data is not too great, what you are seeking to do is easily accomplished using a server job.

As far as I am aware there are not third party tools for maintaining lookups in near real time.

The SCD stage in the next ("Hawk") release will help.