DSXchange

Source Stage (OCI/ODBC) Select ... From SourceTable Where TimeColumn > (select max(LastExtracted) from staging_control_table) @DATE contains the date/time the job was started and will update the LastExtracted date when the job finishes. I would use an aggregator stage to save this value and an upda...

Peter What is Iterations? Hi Kim, my, someone at Ascential is not doing their job, you being a DataStage person should have had someone tell you about Iterations. It is the development methodology first developed by Prism Solutions and it came into Ardent when Ardent bought Prism. It is THE best me...

Most EDWs are seriously lacking in documentation. I want to post an ETL checklist of sample ETL documents like source to target documents, design ideas, how to use metadata to help you design ETL and similar concepts. I just want to help jumpstart this part of our industry. I hate documenting stuff...

Source Stage (OCI/ODBC) Select ... From SourceTable Where TimeColumn > (select max(LastExtracted) from staging_control_table) @DATE contains the date/time the job was started and will update the LastExtracted date when the job finishes. I would use an aggregator stage to save this value and an upda...

Hi All, oh joy oh joy. :D I'm doing a DS project where ALL the source tables actually have a timestamp for their last update and they are small enough to be able to extract from the source oracle database using a where clause on the timestamp. This is the first time ever for me... Hi All, just thin...

Hi Craig, times like this I wish that we would have a place for people to publish/share DS code.......because I'd really like to see how you have done this..... (if you are feeling generous peter@peternolan.com ;-) ) When you say you have used the methodology quoted above, did you mean my suggestion...

Lastly, just by way of contributing to the group.....mistakes I have made... We had some very strange symptoms as we set up our ODBC drivers yesterday. We could connect to a data source and get a list of tables but when we tried to 'view data' we got the dreaded 'DSN not found and no default specifi...

Hi All, I have a table where the column is just defined as 'number'. When the OCI reads in the metadata it sets it to number 38. But I believe that Oracle number can be 126 long. Is this correct? The job fails with a message of oci retrieved truncated data when there are more than 50 such warnings. ...

Hi All, oh joy oh joy. :D I'm doing a DS project where ALL the source tables actually have a timestamp for their last update and they are small enough to be able to extract from the source oracle database using a where clause on the timestamp. This is the first time ever for me... So, I was wonderin...

Very interesting, Peter! I'll have to check it out when I get a chance and see how #2 compares to Uncle Ken's Magnum Opus, which we are currently using with great success. :wink: I'd be interested to hear of anyone else who has done something better with scheduling and restarting DS jobs... I know ...

Hi All, (By way of contributing to the group...) In the world of unix/win based systems and ETL (and to some extent DataStage) I've been frustrated by the lack of control schedulers give me. (Just call me an ex-JCL person where you have 110% control over what is happening and restarts.) As ever, if ...

Hi All, thanks for the input, I'm playing around with a few ways....and I'll get Tom to try out some of your other suggestions on Saturday (that's a working day here...) Currently I have a C++ program working that scans the input folders, performs the decoding, zips, saves, has a wait for file semap...

Hi All, I was wondering if anyone had any hints about how to do this. We will have large numbers of Call Detail Record files coming at us each day. We are getting a file per switch each time a switch fills up it's 12MB file. We will put the files for each switch type into it's own directory. We have...

There are no data types in hashed files. Since integers can have up to 11 characters (including sign), you need to allow 10 bytes, not 4, for a positive integer. This will explain some of the variation. . Hi Ray, thanks for this...I had not guessed that integers were stored as strings inside hash f...

Remember, a hash file is a presized parking lot where the stalls or spots are big enough for a couple of cars each. With a minimum modulus the hash file will always be a certain size when empty. 1. If you remember, I said earlier that it pages the hash file, so that the maximum cache size is how mu...