DSXchange

The compound should be the business key, though a more efficient fact table would be constructed by amalgamating these into a single column, perhaps with delimiter characters. The Surrogate Key must ... In the fact table, only person_id is there. No start date is needed. So this is not a issue at a...

Some of us knew it's a bad data model, but it's built for years and built by paying big bucks by "expert" consultant. That's not the all yet. There are others even worse. Here I just care whether SCD stage apply to this data model or not and try to understand SCD stage: what model it can h...

Thanks ray.wurlod So it can be marked as below: PERSON_PARTY_ID Business Key? INDIVIDUAL_NAME_START_DT Effective date GIVEN_NAME type 2 PREFERRED_NAME type 2 MIDDLE_NAME type 2 FAMILY_NAME type 2 NAME_PREFIX_TXT type 2 NAME_SUFFIX_TXT type 2 INDIVIDUAL_NAME_END_DT Expiration date Also (INDIVIDUAL_NA...

Hi there, Below is a type 2 slow change dimension table: CREATE TABLE INDIVIDUAL_NAME ( PERSON_PARTY_ID INTEGER NOT NULL, INDIVIDUAL_NAME_START_DT DATE NOT NULL, GIVEN_NAME VARCHAR(40) NOT NULL, PREFERRED_NAME VARCHAR(40), MIDDLE_NAME VARCHAR(40), FAMILY_NAME VARCHAR(40), NAME_PREFIX_TXT VARCHAR(10)...

Thanks, ArndW. Here is another approach with better performance: For issue 1: some body said using wc to calculate the number of rows. Such as wc -l ./*.csv|awk 'BEGIN {print "count\tfilename"} {printf "%d\t|%s\n", $1,$2}'. I don't like this approach because it needs to read the ...

Hi there, how can I write a job to handle, like to say, thousand files in a folder. We don't want to process all these files in one batch, but want to each run to limit to process 10 million records in these files. Suppose the total records of these files exceed 100 millions. With this job, two we n...

Thanks ray.wurlod, 8.7 does not support variable length of records. Only fixed length of record. Then the length of a record is fixed. You don't have choice. The length of the record is calculated from its columns. Why is there for you to type in the length? That's a drawback of 7.5. What's the reas...

Hi there,

In the record length field of a sequential file stage, only one choice: fixed. How can variable be chosed? When the empty is chosen, it's changed back to fixed. Except fixed, what can be chosen?

Thanks,

Thanks, ray.wurlod. Do we talk the same thing here? I have a gut feeling we don't.

Very good point, vmcburney, I like this, I'll add it to the article for an approach of restartable ETL job, many thanks. But CDC is only used to handle slow change dimension table. If it's used for other tables, such as fact tables, the performance could be unbearable, unless your fact table is smal...

Okay, that sounds complicated. Absolutely, designing restartable ETL job is a very sophisticated and difficult issue. It's worth an entire chapter of a book to address it, if not a book dedicated to it. Here is an article on it: www.uiis.net/etl/index.php. Any comment and feedback is appreciated. Th...

My "pick up where it left off" comment was specifically directed to ETL jobs, not at the job control level. It may not be typical but it can certainly be done. ... That's interesting, very interesting. Let's look at an example for me to understand how you implement '"pick up where it...

Yes, it's workflow in Informatica, thanks for correcting. Does "it picks up where it left off" mean the same as "it restarts from the failed point"? It's only a part of control job restartability, another part is "it restarts from the designated point". This one is hard...

These tools are used often, a button would be good for the product.

Thanks,