Count the rows in a file

bachi · Post by **bachi** » Thu Dec 03, 2009 4:12 am

I want to split the file in to 2 before my job starts; example if I get 100 records 50 in to one another 50 in to other.Also how can count the records in a file.

thanks

Sreenivasulu · Post by **Sreenivasulu** » Thu Dec 03, 2009 5:11 am

Run unix command 'wc -l' using 'executecommand activity' stage on the file.

Regards
Sreeni

bachi · Post by **bachi** » Thu Dec 03, 2009 6:27 am

Then..

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Thu Dec 03, 2009 6:43 am

bachi wrote:Then..

Then you try it and let us know where you get to or have any issues.

chulett · Post by **chulett** » Thu Dec 03, 2009 7:36 am

There's always the UNIX split command. Out of curiousity, why do you feel the need to split your input files in two?

ray.wurlod · Post by **ray.wurlod** » Thu Dec 03, 2009 2:33 pm

Don't worry about counting anything. Specify two readers per node to read the file. DataStage will work out the 50% point automatically.

chulett · Post by **chulett** » Thu Dec 03, 2009 3:02 pm

Still seems to me that all depends on exactly why they feel the need to split their files in two.

bachi · Post by **bachi** » Fri Dec 04, 2009 1:01 am

only because of huge amount of data

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Fri Dec 04, 2009 2:58 am

Can we get to facts rather than relative terms.

How huge is the data file ?
What is your machine spec ?
What are you trying to achieve (for which you thought splitting the file may be the solution) ?

chulett · Post by **chulett** » Fri Dec 04, 2009 8:30 am

100 records doesn't sound all that huge to me.

Server can easily process millions of records in a single file, can't imagine PX would have any issues but it can depend on exactly what you're doing. I would also be curious what issues you are facing with your large files, perhaps a better solution can be crafted rather than chopping the input files up.