process big data
Posted: Mon Oct 19, 2015 2:05 am
Hi,
I meet a problem that the sequential job process too long time.
there are many Parallel jobs to run in order, as the input files' size are very big, that cause the temporary files so big that the job require long time to read/write.
In such condition, what should I do to tune the job so that the job can process faster ??
I am thinking about combining the parallel jobs to 1 parallel job so that i can recude the read/write. but this is not good idea.
Please kindly advise me how to do. Thanks experts.
walter/
I meet a problem that the sequential job process too long time.
there are many Parallel jobs to run in order, as the input files' size are very big, that cause the temporary files so big that the job require long time to read/write.
In such condition, what should I do to tune the job so that the job can process faster ??
I am thinking about combining the parallel jobs to 1 parallel job so that i can recude the read/write. but this is not good idea.
Please kindly advise me how to do. Thanks experts.
walter/