DSXchange

All,
In my job in the source we have a Folder stage then some transformers to velidate Source file data and then target folder stage.My job is not able to read file of size more than 65 MB.Is there any limitation of the Folder stage with maximum file size it can read or Write.
In my job when ever folder stage encounters a file with size > 65MB it aborts.

Is there any means by which I can over come this limitaion.

Please let me know your suggestions.

Thanks in Advance.
Amit

Dunno of such limitations, except when enforced at the OS level. Ask your UNIX admin about the size constraints.

gateleys

Yes, there are known but undocumented limits on the sizes of files that can be sucked thru a Folder stage. It seems to vary from version to version and O/S to O/S. When you find those limits, as far as I know the only thing you can do about that is ensure that your files are all under that limit - even if it means some kind of preprocessing is in order.

Amit Nema wrote:All,
In my job in the source we have a Folder stage then some transformers to velidate Source file data and then target folder stage.My job is not able to read file of size more than 65 MB.Is there any limitation of the Folder stage with maximum file size it can read or Write.
In my job when ever folder stage encounters a file with size > 65MB it aborts.

Is there any means by which I can over come this limitaion.

Please let me know your suggestions.

Thanks in Advance.
Amit

Hi Amit

I dont think the size of folder will matter in any ways as far as DataStage is concerned.Because,I have tested the folder stage with 90 MB size folder and I met with success.May be problem with O/S.

Regards
Surendra

surendra_ds wrote:I dont think the size of folder will matter in any ways as far as DataStage is concerned.

As I noted, is does matter.

Thanks everyone for your suggestions.We are using Unix at the server end and win XP at the client end.Might be the limitation of Unix.
But through Unix when i try to Copy or move a file with more that 90 MB it works fine.
If any UNIX expert can through some light on this would be of great help.

Thanks,
Amit

Ok... one more time. This is a DataStage limitation, not a UNIX limitation. It is a known issue with the Folder stage.

Open up a support case with Ascential, see if there is any work-around or perhaps a patch they can offer. I'm not aware of one, unfortunately, other than to process smaller files.

Thanks Craig.

No problem. We've seen the same issue on larger XML files and we've had to do the same thing - hack them up into smaller pieces.

If you do pursue this with Ascential and get another approach from them, please post it back here.