External source stage very slow

[chetan.c]

Hi,

I have a n external source stage with below command.

Code: Select all

tar -xOvf /home/LLS2TEALFA01_CSS997.tar

This tar ball has 19k files and around 100,000 records.
The job takes around 30 mins to finish.
The performace of the stage is very slow.I'm just testing the load with the job having only 2 stages: Externalsource--->Dataset.

Can somebody guide me why this is so slow and also how external source stage actually interprets the command and does it wait untill all rows are read to go into next stage?

Thanks,
Chetan.

[ray.wurlod]

How long does the tar command take by itself?

How long does it take to load a Data Set with a comparable number of generated rows?

[chetan.c]

Tar command takes 35 mins on the command prompt.
But when run in the job ,the job is not completeing at all.

Thanks,
Chetan

[qt_ky]

It's not running your tar command in parallel on multiple nodes at once, is it?

[PaulVL]

Also, what exactly are you wishing to pump into your dataset?