Hash File which can hold more than 2 GB data

[abhilashnair]

Well..I just checked the size of DATA.30 file inside the hash file dir...It has crossed 4GB

[abhilashnair]

What perplexes me is that when Stage Write Cache is allowed...hash file can only accomodate 2Gb...when it is not allowed it accomodates more than that...I know it is very slow and the rate at which rows gets populated decreases as time passes...but still it does not abort...

[ray.wurlod]

There is no a priori knowledge of the size of the hashed file when "allow cache" is set at design time. If the hashed file is too big, then a warning is issued, and the hashed file is accessed on disk. But this can not be ascertained until run time.

[abhilashnair]

I just came to know that the OS is limiting hash file size. But the admin is not ready to increase this limit saying that it may affect other hash files which are fine now...Is this really an issue?

[ray.wurlod]

No.

No hashed file can exceed 2GB without being created/resized to use 64-bit addressing internally. Increasing ulimit or a kernel file size parameter will have no impact whatsoever on existing or newly-created hashed files.

[abhilashnair]

We already have 64BIT hash file...But once it reaches 2 GB limit the job aborts...If a 64bIT hashed file can hold data in terabytes..why is this happening?. It is TYPE30 file.

[abhilashnair]

We have a urgent issue in Production...We have 184 million records which have come from source. The size is 18 GB. We want them to be loaded into a hashed file. We have 64BIT Type 30 Hash file. But still after 2Gb it is aborting...Please Help

[chulett]

I just came to know that the OS is limiting hash file size.

This would be your 'urgent' issue, I'd wager. Have your SA remove the file size limitation for your DataStage user.

[ray.wurlod]

If they won't, use a Distributed file of at least ten parts.