DSXchange

Hi folks, starting today, if I try to run jobs from Director, I get a job abortion with SIGINT. Jobs run fine when started in DS Designer. Ideas?

Thanks,
Doug

Hi folks, I noticed that we are getting a growing number of old files in the scratch area. Here's an example: -rw------- 1 dsadm dsdev 2227442 Jul 12 16:49 etleng2node1.0000000000000014.00000000 Shouldn't the scratch area be clear after all the jobs are finished? Perhaps we are accidentally storing ...

the table partition algorithm and the DataStage partition algorithm are not the same. yep, seems to be the limiter. Looks like my "best practice" is to pull the data in parallel without sorting in the db, then hash partition and sort on the hash key in DataStage. My aim is to land all ext...

Hi folks, this might be along the lines of thinking out loud, but it seems to me that I will be using hash partitioning (in general) within DataStage jobs to support (in general) joins. At the same time, it is nice to use ORDER BY in the SQL select statements to reduce the memory burden in the DataS...

Hi folks, it is appropriate and commendable that a site establish coding practices before going into mass production. It is also likely that some sites are new to DataStage. It is therefore likely that someone completely new to DataStage is tasked with establishing the design guide for the developme...

ray.wurlod wrote:If you're using SMP architecture Entire partitioning is essentially free - one copy of the Data Set is emplaced in shared memory and used by all nodes.

Good to know Ray, thanks. It appears that our system is not, uh, utilizing this feature at the moment.

Hi folks, never mind. I figured it out: ALTER TABLE dw_encounters_fact_new DROP CONSTRAINT dw_encounters_fact_new_pk ; DROP INDEX dw_encounters_fact_new_n01 ; DROP INDEX dw_encounters_fact_new_n02 ; DROP INDEX dw_encounters_fact_new_n03 ; DROP INDEX dw_encounters_fact_new_n04 ; DROP INDEX dw_encount...

Hi folks,
I want to drop all the indexes, load the table, and then build all the indexes.

I want to use the oracle Connector before and after SQL scripts to do so.

Does anyone have an example of an oracle script to drop indexes?

Thanks,
Doug

Hi it appears to be possible if the connector is writing. It does not appear to be possible when the connector is reading.

Hi folks, ramping up the number of partitions (two to sixteen) did indeed help considerably. I am now back up to 96K rows/sec. Unfortunately though, running this many partitions affects lookup stages that happen to be using the "entire" partitioning strategy. That required some rework in t...

Hi guys, thanks for your interest. While I am only doing null handling at the moment, there will be column derivation too. There may be other things. So is it appropriate to ask if this is a typical performance hit in a transformer stage?

Hi folks, I've got a 15 column transformation stage, about 400 bytes per row, and I am doing a NullToValue replacement on each column (nothing else). My throughput drops from 108K rows/sec to 58K rows/sec as a result of adding this stage. This seems like a big hit. Are there tuning options, or is th...

You could very easily accomplish the same functionality you are seeking by placing a copy stage in front of your target stage. Hi yes I thought of this after posting. So I envision having a big fat table def that is used to land a subset of the data (via column listing in the select statement) into...

Hi folks, I like the idea of creating table defs from the source database, attaching them to the db input connector, but then using the SQL builder to only select a few columns. I want to build the jobs and downstream stages according to the full table def, not the reduced one, so that if we decide ...

Ernie, again thanks for your help.
Doug