Search found 71 matches

by Yuan_Edward
Sun Jul 15, 2007 6:22 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: uvodbc.config
Replies: 5
Views: 1758

You can specify the service/database details in .odbc.ini. Please refer to the online documentation for details :) Interesting... I''ve always used 'Case 2' without issue regardless of database server location. Thanks Roy and Chulett for your information. Even in case 2, what is the Service ? Is it ...
by Yuan_Edward
Thu Jun 14, 2007 8:54 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: (Server=Dedicated) parameter in TNSNAMES.ORA file
Replies: 4
Views: 6453

Re: (Server=Dedicated) parameter in TNSNAMES.ORA file

In my opinion, (SERVER = DEDICATED) should be there in the TNSNAMES.ORA if the database is set up with DEDICATED Server option. It doesn't affect anything to add that parameter into the TNSNAMES.ORA if it's not there. Hi All, We have a condition here in Data Stage 7.5.2 with server jobs. Currently, ...
by Yuan_Edward
Tue Mar 13, 2007 6:37 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Generating Surrogate Keys via hash functions
Replies: 20
Views: 8020

:cry: I wish there was a perfect hashing algorithm to guarantee uniqueness. That is an issue then. Thanks, Ray. There's one major drawback. It does not guarantee uniqueness. For example CRC32 has about a 1 in 4000000 chance of generating a duplicate. Four million is not that many rows. The entire ra...
by Yuan_Edward
Tue Mar 13, 2007 6:03 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Generating Surrogate Keys via hash functions
Replies: 20
Views: 8020

Hi Craig, thanks for the reply. I am still looking for the suitable function and haven't decided to use CRC32 or MD5 yet. I know the traditional surrogate key is a good choice but just want to get some comments on what will be the drawback of the hashing approach. I know no-one has asked for it, but...
by Yuan_Edward
Tue Mar 13, 2007 5:59 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Generating Surrogate Keys via hash functions
Replies: 20
Views: 8020

Craig, I am still looking for that suitable hashing algorithm and haven't decided to use CRC32 or MD5. I know the traditional surrogate key is a good choice, but just want to know what is the drawback of the hashing approach? Thank you. I know no-one has asked for it, but thought I would throw in my...
by Yuan_Edward
Tue Mar 13, 2007 5:54 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Generating Surrogate Keys via hash functions
Replies: 20
Views: 8020

Hi DSguru, why do you think running numbers is better choice of SKs? I know typical hash functions will return a fixed-length string. I haven't decided which function to use, and I am not sure if hash function could return numbers.
DSguru2B wrote:Likewise Craig. Stick to generation of running number.
by Yuan_Edward
Tue Mar 13, 2007 4:28 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: HOW TO RESTART THE JOB
Replies: 3
Views: 1590

Re: HOW TO RESTART THE JOB

Well, it depends on your requirement and preference. Some of my options will be: 1. truncate and reload the whole 2. Replace/overwrite 3. Take the whole load as a single transaction 4. Do a lookup and only load the rest of records when reloading 5. More... Hi How can I restart the job, if job get ab...
by Yuan_Edward
Tue Mar 13, 2007 3:28 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Generating Surrogate Keys via hash functions
Replies: 20
Views: 8020

Hi Kumar_s, to give you more details, I have a simplified job below: sequencial file =>Transformer=>some additional logic 1=>load into target database (DB 1) ................................|| ................................\/ ..........................Some addtional logic 2 ..........................
by Yuan_Edward
Tue Mar 13, 2007 1:12 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Generating Surrogate Keys via hash functions
Replies: 20
Views: 8020

Good point. Thanks. It should be ok as long as the records in DB1 get SKs allocated. I am assuming that records will be loaded into DB2 (the data quality database) without failure, if they do fail, they have to be fixed and loaded into DB2 again until success...that means, all exceptions will be loa...
by Yuan_Edward
Tue Mar 13, 2007 12:24 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Generating Surrogate Keys via hash functions
Replies: 20
Views: 8020

That's the magic of hash functions. If I apply the same function in two places (one in the ETL flow of loading DB1 and the other in the ETL flow of DB2), I would expect I get the same SKs for the same NKs for both. SK is just an additional key apart form the natural key which are created for some pu...
by Yuan_Edward
Mon Mar 12, 2007 11:50 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Generating Surrogate Keys via hash functions
Replies: 20
Views: 8020

:) Yes, CRC/MD5 is not designed for SKs but it is fit for this purpose, so I am "stealing" it. Our project is processing huge volumnes of data so I am trying to avoid that kind of lookup. And i think hash functions will be generic and easy to use everywhere. Though these functions were not...
by Yuan_Edward
Mon Mar 12, 2007 10:45 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Generating Surrogate Keys via hash functions
Replies: 20
Views: 8020

Generating Surrogate Keys via hash functions

My job is to load flat files into the target database (DB1 for short). Some records will have data quality issues, e.g. where a mandatory field has no vaule, but will be corrected and still get to load into the database. At the same time these records are required to be captured and loaded into a se...
by Yuan_Edward
Thu Mar 08, 2007 4:55 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Transformer logic
Replies: 6
Views: 3723

Re: Transformer logic

The function IsValid can be used to check if a value is numeric, I am using Transformer stage variable to keep the result and then it can be used in the constraint condition. Hi Guys I have to do a some sort of validation process in the datastage transformer. The logic to be used is=> If the input v...
by Yuan_Edward
Thu Mar 08, 2007 4:51 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: type conversion
Replies: 8
Views: 2302

Re: type conversion

That means StringToTimestamp fails. Check the format of the input value where you got the output **********. Hi- I have one funny thing going on with my job. One of the fields in my job needs string to timestamp conversion for that I used the fallowing function. StringToTimestamp((DSLink5.SrcFileDt)...
by Yuan_Edward
Thu Mar 01, 2007 4:56 pm
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Reading a Sequential File
Replies: 14
Views: 5513

You will need to set up an additional property for variable-length fields for fixed-width files. Double click each column in the Column page of the sequencial file stage and in the popup window Edit Column Meta Data set the Field Width property for each of the fields. It should work for all data typ...