Hello,
We have huge files (approx 48) each with a size of around 2GB. We are trying to read these files in a single job using 4 separate Sequential Stages and each stage reading around 12 files.
The job when executed takes too long to run but never finishes.
We tried multiple options of Config File and No. of readers per Node but somehow we do not see improvement in the overall job execution.
We even tried to split it in multiple jobs and it executes fine only when we have less than 5 files in one job. Anything above 5 files in a single job then the job does not complete and stays in Running mode itself.
Kindly let me know if any pointers. Thank You !!!
Issue in reading multiple files
Moderators: chulett, rschirm, roy
How long is too long?
Are you sure it never finishes? (Never hasn't arrived yet.) So, how long did you wait for the job to finish?
Delimited or fixed width files?
What else happens in your job design besides reading files?
Are you sure it never finishes? (Never hasn't arrived yet.) So, how long did you wait for the job to finish?
Delimited or fixed width files?
What else happens in your job design besides reading files?
Choose a job you love, and you will never have to work a day in your life. - Confucius