Hi,
I have job where it reads from a txt file. One of the column in the file is xml.
the file is 15gb and has 40 million records. The job runs for 3 hours.
Is there any way i can improve its performance?
Regards,
Samyam
xml stage performance issues
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 258
- Joined: Tue Jul 04, 2006 10:35 pm
- Location: Toronto
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 258
- Joined: Tue Jul 04, 2006 10:35 pm
- Location: Toronto
Hi All,
Sorry about the delayed response.
We tried a lot of options and the one of them gave us a good performance improvement.
We Split the input file into 4 files of the size 4GB each and triggered the same job 4 times in parallel reading the the 4 different files.
It came down to 1 hour processing time.
Regards,
Samyam
Sorry about the delayed response.
We tried a lot of options and the one of them gave us a good performance improvement.
We Split the input file into 4 files of the size 4GB each and triggered the same job 4 times in parallel reading the the 4 different files.
It came down to 1 hour processing time.
Regards,
Samyam
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: