Sequential File size limit

opdas · Post by **opdas** » Wed Mar 15, 2006 2:21 am

Hi,
Is there any limit to the file size that I can read using sequential file stage using read method as Specific file OR is there any limit to the number of files/file size when I user read method as File Pattern?

Regards
Om

ArndW · Post by **ArndW** » Wed Mar 15, 2006 2:45 am

DataStage doesn't have one, but many file systems default to a 2Gb limit for sequential files. If your OS or installation doesn't have a limit then DataStage doesn't have one, either.

opdas · Post by **opdas** » Wed Mar 15, 2006 2:51 am

ArndW wrote:DataStage doesn't have one, but many file systems default to a 2Gb limit for sequential files. If your OS or installation doesn't have a limit then DataStage doesn't have one, either. ...

Thanks a lot or the reply....
I'm strugling at one more point ..when I use read method as file pattern and I want the name of the file also, we use File column Name but what we get is the File pattern what we pass.

Suppose we pass "/dir/filepattern*.txt" in File Pattern we get the same thing back in the File Name column and not the file name the data being sourced from.

ray.wurlod · Post by **ray.wurlod** » Wed Mar 15, 2006 5:46 am

Beware that your ulimit setting may impose a file size limit.

opdas · Post by **opdas** » Thu Mar 16, 2006 2:06 am

How do I get the file names? Do I have to do some changes in setings?
Coz I get File Column Name as "/dir/filepattern*.txt" the same thing I pass in File Pattern.

kumar_s · Post by **kumar_s** » Thu Mar 16, 2006 2:11 am

opdas wrote:How do I get the file names? Do I have to do some changes in setings?
Coz I get File Column Name as "/dir/filepattern*.txt" the same thing I pass in File Pattern.

If you just want to get the file name, just pass the path name alone, it will read all the filename present in the directory.

opdas · Post by **opdas** » Thu Mar 16, 2006 2:21 am

kumar_s wrote:
opdas wrote:How do I get the file names? Do I have to do some changes in setings?
Coz I get File Column Name as "/dir/filepattern*.txt" the same thing I pass in File Pattern.
If you just want to get the file name, just pass the path name alone, it will read all the filename present in the directory.

I have tried passing just the path name as "/dir/" but thats not even trying to read the content of the file leave alone reading the file name.

kumar_s · Post by **kumar_s** » Thu Mar 16, 2006 2:50 am

A workaround.

Code: Select all

ls -1 /dir/filepattern*.txt > xxx.txt; cat /dir/filepattern*.txt >> xxx.txt

Add this in before job subroutine.
and read xxx.txt as 'specific file'. Now the first line of the xxx.txt will have the filename. In transformer @INROWNUM = 1 can be used to grasp the first line.