SCD typeII implementation

Puli · Post by **Puli** » Fri Jan 30, 2009 6:28 am

I need to Use SCD stage in data stage parallel 8. in my design, I have a source table which lookup dimension table based on primary key, and tgt table have startdate,end_date along with the Src table.

inside the SCD stage there are options to generate surrogate key using flat file option.I need to know whether this flatfile is created at runtime or do i need to create it manually before using the file. and if so how to create it?

there are some flat file created already,which is there in the repository(No idea how that was created), when i try to use that file,the SCD stage is not reading the particular flatfile. am getting the error as:

APT_CombinedOperatorController(0),0: Lookup table is empty, no further warnings will be issued.
APT_CombinedOperatorController(0),0: Unable to read state file /home/dsadm/dsinputfiles/s_key_src.csv: Invalid argument.
Unable to read state file /home/dsadm/dsinputfiles/s_key_src.csv: Invalid argument.

one more doubt: regarding state file in SCD stage...wat are statefiles and how they created...

Mike · Post by **Mike** » Fri Jan 30, 2009 7:28 am

All answered in the product documentation. In particular study the Surrogate Key Generator stage and the Slowly Changing Dimension stage.

The Surrogate Key Generator stage produces a state file... basically the metadata that keeps track of the surrogate key value.

The Slowly Changing Dimension stage is one stage that can utilize the state file for generating surrogate keys. It is a very powerful stage that makes slowly changing dimension logic virtually trivial to implement.

Mike

nani0907 · Post by **nani0907** » Fri Jan 30, 2009 7:59 am

Hi Mike.

I have Console for IBM Information Server installed on my machine. could you please tell me where exactly i can find the product documentation

JRodriguez · Post by **JRodriguez** » Fri Jan 30, 2009 8:46 am

Hi,

Start --> Programs --> Information Server --> Documentation

The IBM Red book "IBM InfoSphere DataStage Data Flow and Job Design" explain this in detail

Mike · Post by **Mike** » Fri Jan 30, 2009 8:47 am

From a typical client install, you would just navigate to it from the Windows Start menu... otherwise you could always go back to your installation media for it.

Mike

nani0907 · Post by **nani0907** » Fri Jan 30, 2009 10:18 am

Thanks alot mike and JRodriguez for your support and help

Puli · Post by **Puli** » Sat Jan 31, 2009 4:16 am

hi Mike,

thanks for ur comments,

if i seperately generate surrogate key using SKG stage and stores them in a file,so that i can refer them as a statefile inside the SCD stage? is it wat u r mentioning....if so, how DS keeps updating the surrogate key when it is in use...

Puli · Post by **Puli** » Sat Jan 31, 2009 4:18 am

hi Mike,

thanks for ur comments,

if i seperately generate surrogate key using SKG stage and stores them in a file,so that i can refer them as a statefile inside the SCD stage? is it wat u r mentioning....if so, how DS keeps updating the surrogate key when it is in use...

Mike · Post by **Mike** » Sat Jan 31, 2009 8:26 am

I'm not concerned with how Datastage manages the state file... just that it does manage it automatically. I haven't tested thoroughly enough to see how it handles concurrent key generating processes, but that's not a concern for me. I use a state file per table and only one job to generate keys for a given table.

Mike

ray.wurlod · Post by **ray.wurlod** » Sat Jan 31, 2009 1:11 pm

Good advice. Treat the state file as a "black box". It works - be happy with that.