Single,variable length seq file to be read in parallel
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 38
- Joined: Tue Jun 19, 2012 11:03 pm
- Location: India
Single,variable length seq file to be read in parallel
Hi,
I have a single sequential file which I want to read in parallel. How can I achieve that? I went through multiple previous posts on the same topic but could not find a solution. There are many contradiciting answers in many posts.
I have a single sequential file which I want to read in parallel. How can I achieve that? I went through multiple previous posts on the same topic but could not find a solution. There are many contradiciting answers in many posts.
-
- Premium Member
- Posts: 38
- Joined: Tue Jun 19, 2012 11:03 pm
- Location: India
Re: Single,variable length seq file to be read in parallel
by making read method as file pattern or by increasing of readers
-
- Premium Member
- Posts: 38
- Joined: Tue Jun 19, 2012 11:03 pm
- Location: India
Re: Single,variable length seq file to be read in parallel
Hi,
Can you please elaborate a bit?
I have heard that number of readers per node can be set only for fixed length files.
Also, please elaborate the first option you have specified.
Can you please elaborate a bit?
I have heard that number of readers per node can be set only for fixed length files.
Also, please elaborate the first option you have specified.
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
Re: Single,variable length seq file to be read in parallel
The first option is not applicable in your case. when you have multiple files to be read and all share a specific pattern, only then you can go ahead with file pattern which is not in your casesharmabhavesh wrote:Also, please elaborate the first option you have specified.
- Zulfi
There's nothing stopping someone from using a file pattern to read a single file. And I believe you may find that the 'only for fixed-width files' mulitple readers advise is for older versions and that restriction was lifted in newer versions. Hopefully someone can confirm / deny...
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
That's good to know... didn't realise it had changed. Although your source file mustn't have a header row as it doesn't like them with multiple readers...chulett wrote: And I believe you may find that the 'only for fixed-width files' mulitple readers advise is for older versions and that restriction was lifted in newer versions. Hopefully someone can confirm / deny...
-
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
-
- Premium Member
- Posts: 38
- Joined: Tue Jun 19, 2012 11:03 pm
- Location: India