Hash File which can hold more than 2 GB data

abhilashnair · Post by **abhilashnair** » Wed Jul 18, 2007 12:03 am

I have data larger than 2GB to be stored into a hash file. The hash file I am using is 64bit. Still it is not able to store the data and the server job which does this aborts with WriteHash() error.

Should I go for a PX job with datasets, or is there any way for making the hash file store that much data. I am talking of 18GB data here.

My job looks like this

ODBC Stage -> Transformer-> Hash File

Maveric · Post by **Maveric** » Wed Jul 18, 2007 12:34 am

Search for "hash file size limit". See if one of those posts help.

ArndW · Post by **ArndW** » Wed Jul 18, 2007 12:50 am

If you have a 64bit hashed file then it will be able to store 18Gb of data. If your OS does not allow files larger than 2Gb then using a 64bit hashed file won't help; and this is what I think has happened on your system.
If you have PX installed and implemented it might make more sense to use datasets for this - but that depends upon how you intend to use the file.

abhilashnair · Post by **abhilashnair** » Wed Jul 18, 2007 2:09 am

ArndW wrote:If you have a 64bit hashed file then it will be able to store 18Gb of data. If your OS does not allow files larger than 2Gb then using a 64bit hashed file won't help; and this is what I think has happ ...

I thought that the max limit for 64bit hash file is not more than 4GB.
BTW..why won't the OS allow the size of 18GB...I am not clear on that.

ArndW · Post by **ArndW** » Wed Jul 18, 2007 2:22 am

If you do the math, a 64bit pointer can address significantly more than 4Gb. Just call up your handy calculator and try 2^64.
I never said that the OS won't allow 18Gb, that was your interpretation. Historically many OS's limited files to 2Gb, with the advent of 64bit addressing they allow files to grow, but on some you need to enable that feature from the OS, and I assume that is what has happened in your case, if you actually did create a 64bit hashed file.

abhilashnair · Post by **abhilashnair** » Wed Jul 18, 2007 3:39 am

Ok. Now I get your point ArndW. We have DS set up on Unix. Any idea how do we go about checking whether the OS allows this much data...or is it something for the Unix Geeks to answer?

Cr.Cezon · Post by **Cr.Cezon** » Wed Jul 18, 2007 3:56 am

You must do ulimit -a for the user root to see the kernells parameters. You can put it to ulimited, then your os will have not this limitation.

Also,You can resize your file hash to increase it up to 2GB using the command RESIZE. but a Hash file more than 2GB is not trateable in goog performance.You willl have problems to work with it.

If you have PX intalled it is better to use DataSets.

Cristina.

abhilashnair wrote:Ok. Now I get your point ArndW. We have DS set up on Unix. Any idea how do we go about checking whether the OS allows this much data...or is it something for the Unix Geeks to answer?

abhilashnair · Post by **abhilashnair** » Wed Jul 18, 2007 4:20 am

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited

The above is the output of ulimit -a in the directory where my hash file is present

Cr.Cezon · Post by **Cr.Cezon** » Wed Jul 18, 2007 4:30 am

have you done these command as root user?

abhilashnair wrote:time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited

The above is the output of ulimit -a in the directory where my hash file is present

abhilashnair · Post by **abhilashnair** » Wed Jul 18, 2007 4:35 am

I logged in with my own id and executed this command. I am not aware what exactly do you mean by root user?

Cr.Cezon · Post by **Cr.Cezon** » Wed Jul 18, 2007 4:38 am

abhilashnair wrote:I logged in with my own id and executed this command. I am not aware what exactly do you mean by root user?

root is the user administrator of your unix machine.

The ulimit -a for you user is not relevant because DataStage don't use it.

abhilashnair · Post by **abhilashnair** » Wed Jul 18, 2007 4:41 am

Oh! Then we have an issue here. I have to contact Unix admin. He is in the US and myself in India.

Cr.Cezon · Post by **Cr.Cezon** » Wed Jul 18, 2007 4:49 am

abhilashnair wrote:Oh! Then we have an issue here. I have to contact Unix admin. He is in the US and myself in India.

then ask him, but DataStage has another limitation with Hash file that is not in the s.o.
for hash files greater than 2 GB the performance is not good.

If you want to work with it you must RESIZE the hash file after you have created but the performance is not good.
Better use other stage

abhilashnair · Post by **abhilashnair** » Wed Jul 18, 2007 5:15 am

I just tried running the same DS job with 'Allow Stage Write Cache' Option unchecked. It was checked earlier when the job aborted.

The job is running fine. Now the point where this job earlier aborted has been passed and the job is still running...

But it is very slow....

Cr.Cezon · Post by **Cr.Cezon** » Wed Jul 18, 2007 5:20 am

yes, the performace will be not very good, and when you will reach the limit of 2GB (or if you have 64 bits upper) the job will abort .....