Size limit for Folder Stage

Amit Nema · Post by **Amit Nema** » Tue May 23, 2006 11:23 am

All,
In my job in the source we have a Folder stage then some transformers to velidate Source file data and then target folder stage.My job is not able to read file of size more than 65 MB.Is there any limitation of the Folder stage with maximum file size it can read or Write.
In my job when ever folder stage encounters a file with size > 65MB it aborts.

Is there any means by which I can over come this limitaion.

Please let me know your suggestions.

Thanks in Advance.
Amit

gateleys · Post by **gateleys** » Tue May 23, 2006 11:38 am

Dunno of such limitations, except when enforced at the OS level. Ask your UNIX admin about the size constraints.

gateleys

chulett · Post by **chulett** » Tue May 23, 2006 12:44 pm

Yes, there are known but undocumented limits on the sizes of files that can be sucked thru a Folder stage. It seems to vary from version to version and O/S to O/S. When you find those limits, as far as I know the only thing you can do about that is ensure that your files are all under that limit - even if it means some kind of preprocessing is in order.

surendra_ds · Post by **surendra_ds** » Wed May 24, 2006 2:04 am

Amit Nema wrote:All,
In my job in the source we have a Folder stage then some transformers to velidate Source file data and then target folder stage.My job is not able to read file of size more than 65 MB.Is there any limitation of the Folder stage with maximum file size it can read or Write.
In my job when ever folder stage encounters a file with size > 65MB it aborts.

Is there any means by which I can over come this limitaion.

Please let me know your suggestions.

Thanks in Advance.
Amit

Hi Amit

I dont think the size of folder will matter in any ways as far as DataStage is concerned.Because,I have tested the folder stage with 90 MB size folder and I met with success.May be problem with O/S.

Regards
Surendra

chulett · Post by **chulett** » Wed May 24, 2006 6:40 am

surendra_ds wrote:I dont think the size of folder will matter in any ways as far as DataStage is concerned.

As I noted, is does matter.

Amit Nema · Post by **Amit Nema** » Wed May 24, 2006 11:54 am

Thanks everyone for your suggestions.We are using Unix at the server end and win XP at the client end.Might be the limitation of Unix.
But through Unix when i try to Copy or move a file with more that 90 MB it works fine.
If any UNIX expert can through some light on this would be of great help.

Thanks,
Amit

chulett · Post by **chulett** » Wed May 24, 2006 12:23 pm

Ok... one more time. This is a DataStage limitation, not a UNIX limitation. It is a known issue with the Folder stage.

Open up a support case with Ascential, see if there is any work-around or perhaps a patch they can offer. I'm not aware of one, unfortunately, other than to process smaller files.

Amit Nema · Post by **Amit Nema** » Wed May 24, 2006 11:10 pm

Thanks Craig.

chulett · Post by **chulett** » Thu May 25, 2006 6:43 am

No problem. We've seen the same issue on larger XML files and we've had to do the same thing - hack them up into smaller pieces.

If you do pursue this with Ascential and get another approach from them, please post it back here.

DSXchange