Creating Hashfiles: size limitations
Moderators: chulett, rschirm, roy
Creating Hashfiles: size limitations
I have to load 40 million records into a hashfile. The size of each row in the Hashfile is 50 bytes.
When I created it as dynamic hashfile it failed after a million rows with the error:
Abnormal termination of stage CopyOfint2outLoadAddrProcResHash..LoadHashCodedRecordsOut.IDENT1 detected
So I moved to the static hashfile. Used alogrithim 16(ascii key, any variations), created 4500 buckets each of size 8196K. The job failed with the error.
CopyOfint2outLoadAddrProcResHash..LoadHashCodedRecordsOut.IDENT1:|CopyOfint2outLoadAddrProcResHash..LoadHashCodedRecordsOut.LoadTableFile: DSD.UVOpen mkdbfile: unable to create a 32-bit file greater than 2 gigabytes.
.|
I have a couple of questions:
1. Is there a limit of the size of the Hashfiles. I thought the 2GB size no longer existed.
2. Could you suggest approaches, of How i could load 40 millions rows into a hashfile.
When I created it as dynamic hashfile it failed after a million rows with the error:
Abnormal termination of stage CopyOfint2outLoadAddrProcResHash..LoadHashCodedRecordsOut.IDENT1 detected
So I moved to the static hashfile. Used alogrithim 16(ascii key, any variations), created 4500 buckets each of size 8196K. The job failed with the error.
CopyOfint2outLoadAddrProcResHash..LoadHashCodedRecordsOut.IDENT1:|CopyOfint2outLoadAddrProcResHash..LoadHashCodedRecordsOut.LoadTableFile: DSD.UVOpen mkdbfile: unable to create a 32-bit file greater than 2 gigabytes.
.|
I have a couple of questions:
1. Is there a limit of the size of the Hashfiles. I thought the 2GB size no longer existed.
2. Could you suggest approaches, of How i could load 40 millions rows into a hashfile.
-
- Premium Member
- Posts: 385
- Joined: Tue Oct 07, 2003 4:55 am
HFC - the very useful but 'unsupported' Hash File Calculator utility, written by our dear friend Mr Wurlod. As noted, it will generate the create statement needed based on your input parameters.ariear wrote:There you'll find a utility called something like HFR.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
64-bit refers only to the internal forward and backward pointers within the file structure. Therefore, it does not matter that DataStage is a "32-bit application" (I'm not sure what that means - is it 16 times better than a "two-bit application"?). However, it is essential that the Operating System can support 64-bit quantities for this to work.
Incidentally, it's the 32-bit pointers that cause the 2GB limit. Observe that 2^31 is 2GB, and take away the sign bit from the pointer, and that leaves 31 bits with which to determine an offset.
Incidentally, it's the 32-bit pointers that cause the 2GB limit. Observe that 2^31 is 2GB, and take away the sign bit from the pointer, and that leaves 31 bits with which to determine an offset.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Not certain, but I believe so for current version.
There is a list of platforms somewhere in the UniVerse manuals for which 64-bit files are supported; I guess you could compare this to the list of supported platforms for DataStage.
Some of the older versions of DataStage and operating systems may not support 64-bit pointers.
If anyone wants to test a particular system all you have to do is to try to create a hashed file (or UV table) with 64-bit option set. If it fails because the O/S doesn't support them, the diagnostic message will say so.
There is a list of platforms somewhere in the UniVerse manuals for which 64-bit files are supported; I guess you could compare this to the list of supported platforms for DataStage.
Some of the older versions of DataStage and operating systems may not support 64-bit pointers.
If anyone wants to test a particular system all you have to do is to try to create a hashed file (or UV table) with 64-bit option set. If it fails because the O/S doesn't support them, the diagnostic message will say so.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.