I am working with DataStage Version 7.5.1.A and i am expieriencing a significant Performance loss while using Load PACK stage ( ie it takes a lot of time to complete the Job).
Can anyone share his experience of working with LOAD PACK stage, the performance that load pack gave and anything that would be useful ( if possible in terms of Rows per second). :D
And any Inputs regarding Load Pack is welcome .
Currently we are creating Flatfiles in Datastage and then these flatfiles are SFTPed to the SAPBW server and after that ,jobs in BW populate their data targets using these Flatfiles (.csv file).
The whole thing , ie the data to go from oracle Database to BW data Targets takes 2.5 Hrs.
I thought may be by using SAPBW load Pack we can reduce this Time .
But it seems Load pack is also taking that much time.
Any inputs are Highly appreciated...
How is the information currently loading into SAPBW? Is it possible that the actual lag in load time is actually the load process into BW? If so then using the load pack will probably not give much benefit.
currently we are generating flatfiles in datastage and then these files are sftped to the SAPBW server. the SFTP part takes a long time.
Could you plz throw some light on this.
No matter what you do, the same volume of data needs to be transmitted across your network. Perhaps you could consider compressing your flat files prior to transmission, and uncompressing them afterwards.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
You could also raise a red flag on your network performance between the boxes. They might be able to find some extra bandwidth. If DataStage Load pack is doing a bulk load then under the covers it's writing a file to the SAP server which is using it as the load file - so it's doing much the same as your old design but without the scripting steps.
You are on server edition, you could consider a multiple instance job and partitioning your data in half or thirds or quarters and trying to deliver the data across multiple identical jobs, though I'm not sure whether the SAP load would support concurrent loads. You could also try a multiple instance job delivering more than one sequence file and concatenate them together. You still hit the bottleneck of bandwidth to SAP however you might squeeze some extra performance and the concatenation at the end should be very fast.
( ie it takes a lot of time to complete the Job).
.