Write to hash file is taking long time Nearly 13hr

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
manu.dwhds
Participant
Posts: 30
Joined: Fri Sep 26, 2008 9:33 pm
Location: Bangalore

Write to hash file is taking long time Nearly 13hr

Post by manu.dwhds »

Dear DS Guru's ,

I have an issue in production which one of the jobs taking 13hr to write the data into hash file .

Job design :extract the data from Teradata stage and transformer(droping one column out of 4 columns ) and writing 3 columns into hash file .

write mode :clear file before writing

Teradata pulling data within 15 min then writing the hash file is taking more than 13hr .
Record count :814375543

Please help me to tune this job .

Your timely help is very much appreciated .

Regards,
Mano
Manu
SURA
Premium Member
Premium Member
Posts: 1229
Joined: Sat Jul 14, 2007 5:16 am
Location: Sydney

Re: Write to hash file is taking long time Nearly 13hr

Post by SURA »

Which unix flavour is that?
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

What's the key? How big (wide) are the rows? What hashing algorithm have you specified? What size for the hashed file? Get these right and you will be able to populate it faster.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
manu.dwhds
Participant
Posts: 30
Joined: Fri Sep 26, 2008 9:33 pm
Location: Bangalore

Post by manu.dwhds »

Create a file --Not selected hence options disabled to chosse hash method .
Update action --Clear file before writing .
Key column :varchar(255) not null

Please suggest best options to write the data into hash .

Regards,
Mano
Manu
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

manu.dwhds wrote:Create a file --Not selected hence options disabled to chosse hash method.
Or to choose a starting size. So you start off with the smallest possible hashed file and then proceed to write 814 million records to it. It spends the majority of your 13 hours basically resizing and expanding the hashed file.

I'd start by suggesting you find the Hashed File Calculator utility that I assume is still part of the delivery media. Answer a few questions about your hashed file and it will give you values to plug into those 'Create File' options you currently have disabled so that it can be created at a proper size for the volume of data it will be holding.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply