Performance issue while loading into table

monaz · Post by **monaz** » Wed Apr 29, 2009 6:53 am

hi All,

i am facing some performace issue while loading the data into table at the output stage

I am using below stages to load data

Four Datasets are concatenated using funnel stage late it goes to the transformer stage which invokes a external C routine and loads into table

Is there any bulk loading option in DataStage ODBC stage?

I run the job against 150K records, it took almost 3 hours.

The same job i laoded the output into sequential file completed in mins.

Please suggest..

chulett · Post by **chulett** » Wed Apr 29, 2009 7:15 am

Define what 'loading' means here - all inserts? To answer the bulk load question, you'd need to actually mention the name of the database you're working with.

ray.wurlod · Post by **ray.wurlod** » Wed Apr 29, 2009 4:46 pm

Prove that the "external C routine" is efficient (not the bottleneck).

How do you define "performance" in an ETL context?

monaz · Post by **monaz** » Wed Apr 29, 2009 10:09 pm

ray.wurlod wrote:Prove that the "external C routine" is efficient (not the bottleneck).

How do you define "performance" in an ETL context? ...

Thanks to all...

Yes i am using an ODBC stage to Insert or update the records coming from the source datasets into sybase database in parallel job.

Since i have 150k records expected to insert into table everyday so its just taking time around 3hrs to complete.

My question is that is there any option in which we can bulk insert or update the records into table.

Please advice..

chulett · Post by **chulett** » Wed Apr 29, 2009 10:17 pm

You can't do "insert or update" with a bulk loader.