Hash File which can hold more than 2 GB data
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 284
- Joined: Fri Oct 13, 2006 4:31 am
Hash File which can hold more than 2 GB data
I have data larger than 2GB to be stored into a hash file. The hash file I am using is 64bit. Still it is not able to store the data and the server job which does this aborts with WriteHash() error.
Should I go for a PX job with datasets, or is there any way for making the hash file store that much data. I am talking of 18GB data here.
My job looks like this
ODBC Stage -> Transformer-> Hash File
Should I go for a PX job with datasets, or is there any way for making the hash file store that much data. I am talking of 18GB data here.
My job looks like this
ODBC Stage -> Transformer-> Hash File
If you have a 64bit hashed file then it will be able to store 18Gb of data. If your OS does not allow files larger than 2Gb then using a 64bit hashed file won't help; and this is what I think has happened on your system.
If you have PX installed and implemented it might make more sense to use datasets for this - but that depends upon how you intend to use the file.
If you have PX installed and implemented it might make more sense to use datasets for this - but that depends upon how you intend to use the file.
-
- Participant
- Posts: 284
- Joined: Fri Oct 13, 2006 4:31 am
I thought that the max limit for 64bit hash file is not more than 4GB.ArndW wrote:If you have a 64bit hashed file then it will be able to store 18Gb of data. If your OS does not allow files larger than 2Gb then using a 64bit hashed file won't help; and this is what I think has happ ...
BTW..why won't the OS allow the size of 18GB...I am not clear on that.
If you do the math, a 64bit pointer can address significantly more than 4Gb. Just call up your handy calculator and try 2^64.
I never said that the OS won't allow 18Gb, that was your interpretation. Historically many OS's limited files to 2Gb, with the advent of 64bit addressing they allow files to grow, but on some you need to enable that feature from the OS, and I assume that is what has happened in your case, if you actually did create a 64bit hashed file.
I never said that the OS won't allow 18Gb, that was your interpretation. Historically many OS's limited files to 2Gb, with the advent of 64bit addressing they allow files to grow, but on some you need to enable that feature from the OS, and I assume that is what has happened in your case, if you actually did create a 64bit hashed file.
-
- Participant
- Posts: 284
- Joined: Fri Oct 13, 2006 4:31 am
You must do ulimit -a for the user root to see the kernells parameters. You can put it to ulimited, then your os will have not this limitation.
Also,You can resize your file hash to increase it up to 2GB using the command RESIZE. but a Hash file more than 2GB is not trateable in goog performance.You willl have problems to work with it.
If you have PX intalled it is better to use DataSets.
Cristina.
Also,You can resize your file hash to increase it up to 2GB using the command RESIZE. but a Hash file more than 2GB is not trateable in goog performance.You willl have problems to work with it.
If you have PX intalled it is better to use DataSets.
Cristina.
abhilashnair wrote:Ok. Now I get your point ArndW. We have DS set up on Unix. Any idea how do we go about checking whether the OS allows this much data...or is it something for the Unix Geeks to answer?
-
- Participant
- Posts: 284
- Joined: Fri Oct 13, 2006 4:31 am
have you done these command as root user?
abhilashnair wrote:time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited
The above is the output of ulimit -a in the directory where my hash file is present
-
- Participant
- Posts: 284
- Joined: Fri Oct 13, 2006 4:31 am
-
- Participant
- Posts: 284
- Joined: Fri Oct 13, 2006 4:31 am
then ask him, but DataStage has another limitation with Hash file that is not in the s.o.abhilashnair wrote:Oh! Then we have an issue here. I have to contact Unix admin. He is in the US and myself in India.
for hash files greater than 2 GB the performance is not good.
If you want to work with it you must RESIZE the hash file after you have created but the performance is not good.
Better use other stage
-
- Participant
- Posts: 284
- Joined: Fri Oct 13, 2006 4:31 am