Sequential File

PhaniKonagalla · Post by **PhaniKonagalla** » Tue May 29, 2012 2:08 am

Hi,

Can we use UNIX commands in Sequential file stage other than FileName.
Can the Sequential file run in parallel?

ray.wurlod · Post by **ray.wurlod** » Tue May 29, 2012 3:33 am

Yes and yes.

You can use any well-formed UNIX command as a Filter. The Sequential File stage will consume stdout from the filter command.

The Sequential File stage can run in parallel if you specify multiple readers per node or specify that it is to read more than one file.

zulfi123786 · Post by **zulfi123786** » Tue May 29, 2012 3:53 am

you could also read a file parallely by specifying multiple nodes per reader provided its a fixed width file.

zulfi123786 · Post by **zulfi123786** » Tue May 29, 2012 3:57 am

ray.wurlod wrote:The Sequential File stage can run in parallel if you specify multiple readers per node or specify that it is to read more than one file.

When a file pattern is used, it was my assumption that the files are concatenated and the result is read sequentially. with the above quote does it mean the files are read in parallel, if yes then is it one reader per file ?

jwiles · Post by **jwiles** » Tue May 29, 2012 9:02 am

Your assumption is correct as far as the default operation of the stage is concerned when using a file pattern: It will concatenate the files prior to reading them.

The files which match a pattern can be read in parallel if the environment variable <a href="http://publib.boulder.ibm.com/infocente ... FILESET</a> is set. The underlying operator then will read the files as if they were members of a fileset. IIRC it will be one reader per file, up to the degree of parallelism in which your job is running.

Regards,