Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett , rschirm , roy
bachi
Participant
Posts: 28 Joined: Sun May 25, 2008 7:02 am
Post
by bachi » Thu Dec 03, 2009 4:12 am
I want to split the file in to 2 before my job starts; example if I get 100 records 50 in to one another 50 in to other.Also how can count the records in a file.
thanks
Sreenivasulu
Premium Member
Posts: 892 Joined: Thu Oct 16, 2003 5:18 am
Post
by Sreenivasulu » Thu Dec 03, 2009 5:11 am
Run unix command 'wc -l' using 'executecommand activity' stage on the file.
Regards
Sreeni
bachi
Participant
Posts: 28 Joined: Sun May 25, 2008 7:02 am
Post
by bachi » Thu Dec 03, 2009 6:27 am
Then..
Sainath.Srinivasan
Participant
Posts: 3337 Joined: Mon Jan 17, 2005 4:49 am
Location: United Kingdom
Post
by Sainath.Srinivasan » Thu Dec 03, 2009 6:43 am
bachi wrote: Then..
Then
you try it and let us know where you get to or have any issues.
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Thu Dec 03, 2009 7:36 am
There's always the UNIX split command. Out of curiousity, why do you feel the need to split your input files in two?
-craig
"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607 Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Post
by ray.wurlod » Thu Dec 03, 2009 2:33 pm
Don't worry about counting anything. Specify two readers per node to read the file. DataStage will work out the 50% point automatically.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Thu Dec 03, 2009 3:02 pm
Still seems to me that all depends on exactly
why they feel the need to split their files in two.
-craig
"You can never have too many knives" -- Logan Nine Fingers
bachi
Participant
Posts: 28 Joined: Sun May 25, 2008 7:02 am
Post
by bachi » Fri Dec 04, 2009 1:01 am
only because of huge amount of data
Sainath.Srinivasan
Participant
Posts: 3337 Joined: Mon Jan 17, 2005 4:49 am
Location: United Kingdom
Post
by Sainath.Srinivasan » Fri Dec 04, 2009 2:58 am
Can we get to facts rather than relative terms.
How huge is the data file ?
What is your machine spec ?
What are you trying to achieve (for which you thought splitting the file may be the solution) ?
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Fri Dec 04, 2009 8:30 am
100 records doesn't sound all that huge to me.
Server can easily process
millions of records in a single file, can't imagine PX would have any issues but it can depend on exactly what you're doing. I would also be curious what issues you are facing with your large files, perhaps a better solution can be crafted rather than chopping the input files up.
-craig
"You can never have too many knives" -- Logan Nine Fingers