xml performance
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 47
- Joined: Thu Dec 09, 2010 4:37 am
xml performance
HI All
i am extracting the records from 29 oracle tables and creating the 29 xml file then merging the 29 xml file into one xml file through unix script
then zipping the merged file and doing the sftp through unix script
currently my job is taking around 30-40 sec (depends upon the data in table) to generate and sftp the xml
is there any way by which i can reduce the generation time by 10-15 sec or how i can double the throughput
i am extracting the records from 29 oracle tables and creating the 29 xml file then merging the 29 xml file into one xml file through unix script
then zipping the merged file and doing the sftp through unix script
currently my job is taking around 30-40 sec (depends upon the data in table) to generate and sftp the xml
is there any way by which i can reduce the generation time by 10-15 sec or how i can double the throughput
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Are you running the 29 jobs consecutively? Running them simultaneously (or in bunches, if you don't have the capacity for all 29) should cut some processing time.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 47
- Joined: Thu Dec 09, 2010 4:37 am
Convert them to Server jobs.
(in the immortal words of BOC - don't fear the Server)
(in the immortal words of BOC - don't fear the Server)
Last edited by chulett on Wed Sep 14, 2011 7:51 am, edited 1 time in total.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
Absolutely. If they are that simple, chances are they run in less than 1 second, and the overhead is just EE Job Start up time.
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
-
- Participant
- Posts: 47
- Joined: Thu Dec 09, 2010 4:37 am
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
No way to directly convert them, but the methodology is similar, so you shouldn't have too much difficulty, and the syntax of the xmlStage is identical.
Save the output link definition for your xmlInput stage in EE to a tabledef and you can then just "load" that into the output link of your server xmlInput Stage.
You'll still need to manually re-apply various properties in other parts of the Stage and Job.
Ernie
Save the output link definition for your xmlInput stage in EE to a tabledef and you can then just "load" that into the output link of your server xmlInput Stage.
You'll still need to manually re-apply various properties in other parts of the Stage and Job.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
-
- Participant
- Posts: 47
- Joined: Thu Dec 09, 2010 4:37 am
Multiple Instances don't really apply here. There are lots of things you can do with multiple instances, one of them being a 'convenience' ...to have one job design, and run it (say) 15 times concurrently, passing different job parameters to each.
Based on what we've been discussing, going to Server Jobs is going to get you a dramatic improvement for these tiny files...mostly because the processing time for parsing the information is probably not where your bottleneck is.
Ernie
Based on what we've been discussing, going to Server Jobs is going to get you a dramatic improvement for these tiny files...mostly because the processing time for parsing the information is probably not where your bottleneck is.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
I'm about to post an XML performance problem, and it looks like parsing is where our bottleneck is.eostic wrote:...mostly because the processing time for parsing the information is probably not where your bottleneck is.
Ernie
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
Is this still the original thread?...the original discussed writing xml, so it's hard to tell what is being discussed...
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Without muck, there can be no muckraking.chulett wrote:Franklin has his own thread now, let's not muck this one up with his stuff.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872