Hi,
Can we use UNIX commands in Sequential file stage other than FileName.
Can the Sequential file run in parallel?
Sequential File
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 7
- Joined: Tue Jul 28, 2009 6:09 am
- Location: Chennai
Sequential File
Phani Kumar
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Yes and yes.
You can use any well-formed UNIX command as a Filter. The Sequential File stage will consume stdout from the filter command.
The Sequential File stage can run in parallel if you specify multiple readers per node or specify that it is to read more than one file.
You can use any well-formed UNIX command as a Filter. The Sequential File stage will consume stdout from the filter command.
The Sequential File stage can run in parallel if you specify multiple readers per node or specify that it is to read more than one file.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
When a file pattern is used, it was my assumption that the files are concatenated and the result is read sequentially. with the above quote does it mean the files are read in parallel, if yes then is it one reader per file ?ray.wurlod wrote:The Sequential File stage can run in parallel if you specify multiple readers per node or specify that it is to read more than one file.
- Zulfi
Your assumption is correct as far as the default operation of the stage is concerned when using a file pattern: It will concatenate the files prior to reading them.
The files which match a pattern can be read in parallel if the environment variable <a href="http://publib.boulder.ibm.com/infocente ... FILESET</a> is set. The underlying operator then will read the files as if they were members of a fileset. IIRC it will be one reader per file, up to the degree of parallelism in which your job is running.
Regards,
The files which match a pattern can be read in parallel if the environment variable <a href="http://publib.boulder.ibm.com/infocente ... FILESET</a> is set. The underlying operator then will read the files as if they were members of a fileset. IIRC it will be one reader per file, up to the degree of parallelism in which your job is running.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.