Page 2 of 2
Posted: Fri May 20, 2005 6:03 am
by ArndW
mmanes - if it's any consolation, I am getting the same problem now. My disk isn't full and it seems that the error message happens when the file gets just a bit over 1Gb large. I put a display of "ulimit -a" into the job and it shows unlimited and I wrote another test Px job that created a sequential file of 2Gb with no problems.
I think it might be something to do with doing a DataSet -> Lookup stage directly, but can't get rid of the error!
Posted: Fri May 20, 2005 11:04 am
by mmanes
I've solved the problem...
You're usin memory windows?
Posted: Fri May 20, 2005 11:22 am
by ArndW
No, I am having this problem on an AIX box. What is/was the cause - I haven't solved my issues yet.
Posted: Sun May 22, 2005 2:28 am
by elavenil
Can you explain when you are getting this error and what the job design is.
If you use Dataset as a lookup and PX engine will create lookup table while running a job so this error is encountered if the lookup table which you are trying to create exceeds 2GB in size (even though the file size is unlimited for the user).
Check the lookup dataset and use join instead of lookup if you use dataset as a lookup.
Hope TWH.
Regards
Saravanan
Posted: Sun May 22, 2005 5:39 pm
by ray.wurlod
It's my understanding that a persistent DataSet can exist as multiple files, each of which isn't bigger than 2GB. Are you saying that this is not the case for virtual Data Sets?
Posted: Mon May 23, 2005 12:33 am
by elavenil
Hi Ray,
When Dataset is created, it is created multiple disks/single disk based on the nodes that are mentioned in the configuration file.
When the dataset is used as a lookup (instead of lookup file set), PX engine will create lookup table the lookup table exceeds the size that is mentioned for the user, this problem occurred. I had the same problem in my previous project and resolved using join (instead of lookup) to join input and lookup datasets.
Hope this would clarify the question.
Thanks & Regards
saravanan
Posted: Mon May 23, 2005 1:02 am
by ArndW
elavenil,
I don't know if this is my problem, although the description does make sense. In my case the lookup file set is just over 1Gb, but my ulimit is large enough... I've written some test jobs to ensure that my files and datasets can be larger and they work. I will start analyzing the error later on today and hope to find the cause (and the solution) by then.
Posted: Mon May 23, 2005 6:22 am
by mmanes
This solution is for HP.
For more info see my post at
viewtopic.php?t=92775&highlight=
In that configuration u may add the virtual nodes <hostname>, <host1>,... in /etc/hosts
By,
Matteo.