It is a very useful tool, but you need training to be able to use it properly. Hopefully they will not pull the tool because they have not given proper training, that would be the result with any tool, not just DataStage.jamesrender wrote:chulett wrote:
I think that it looks like a really useful tool
Terrible Performance due to log file
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 437
- Joined: Fri Oct 21, 2005 10:00 pm
Keith Williams
keith@peacefieldinc.com
keith@peacefieldinc.com
-
- Participant
- Posts: 13
- Joined: Fri Jan 06, 2006 9:20 am
bit of a chicken and egg situation, I can't convince them to keep it without being proficient with it, and I can't get that proficiency if it goes in the next month.. their feeling is that its an extra level of complexity for the application, will probably want some sql/loader process created, even though that is what DS is all about. any opportunity to save a few bucks by dropping a license..kwwilliams wrote:jamesrender wrote:
It is a very useful tool, but you need training to be able to use it properly. Hopefully they will not pull the tool because they have not given proper training, that would be the result with any tool, not just DataStage.
I'm sure that I'll discover a thorny issue that datastage solved once the decision to drop it has been taken..
Last edited by jamesrender on Mon Jan 16, 2006 10:33 am, edited 1 time in total.
Be careful with your quoting... that wasn't me.kwwilliams wrote:jamesrender wrote:It is a very useful tool, but you need training to be able to use it properly. Hopefully they will not pull the tool because they have not given proper training, that would be the result with any tool, not just DataStage.chulett wrote:
I think that it looks like a really useful tool
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 13
- Joined: Fri Jan 06, 2006 9:20 am
Hi,
Couldnt help asking. Ray you said that keepipng logs of DS for a long time without purge would degrade performance? We keep getting requets of data load dating back to more then 12 months, due to which we dont delete any logs. Would purging them help in performance improvement? Also is there a way where I can store these log files? Outside of DS I mean so that I can do some archiving?
Cheers,
Vishy
Couldnt help asking. Ray you said that keepipng logs of DS for a long time without purge would degrade performance? We keep getting requets of data load dating back to more then 12 months, due to which we dont delete any logs. Would purging them help in performance improvement? Also is there a way where I can store these log files? Outside of DS I mean so that I can do some archiving?
Cheers,
Vishy
-
- Participant
- Posts: 13
- Joined: Fri Jan 06, 2006 9:20 am
I can't speak with any authority, but certainly in my case, the size of the log had a direct impact on job performance. It is possible to save DS logs to a file system. I have done it once, but I can't remember how.. sorry.Viswanath wrote:Hi,
Couldnt help asking. Ray you said that keepipng logs of DS for a long time without purge would degrade performance? We keep getting requets of data load dating back to more then 12 months, due to which we dont delete any logs. Would purging them help in performance improvement? Also is there a way where I can store these log files? Outside of DS I mean so that I can do some archiving?
Cheers,
Vishy
The logs degrade performance only when the job has to put a message into its log file. Since the log file is actually a dynamic hash file, an extremely large hash file may take longer to add a row, especially if the file just happens to need to dynamically expand at that point in time. If the job is logging a lot of messages, then yes, absolutely you will notice a severe impact to runtime performance.
At the start and end of processing, the job logs informational messages. There is an impact to total runtime, so keeping the logs purged allows startup and wrapup processing to be minimal. You will notice two reasons a job finishes moving data, yet still seems to be doing something else before finishing: logging/purging messages and updating the &PH& information. For faster startup/wrapup, purge your log files and keep the &PH& directory in the project clean.
At the start and end of processing, the job logs informational messages. There is an impact to total runtime, so keeping the logs purged allows startup and wrapup processing to be minimal. You will notice two reasons a job finishes moving data, yet still seems to be doing something else before finishing: logging/purging messages and updating the &PH& information. For faster startup/wrapup, purge your log files and keep the &PH& directory in the project clean.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Thanks Kenneth.
I guess I have some work to do then. We have around 300 odd jobs in our Production project. I have to do a manual purge now and the enable auto purge for these.
But is there a way by which I can automatically store these logs in a file? I probably would need them in future for auditing purposes.
Cheers,
Vishy
I guess I have some work to do then. We have around 300 odd jobs in our Production project. I have to do a manual purge now and the enable auto purge for these.
But is there a way by which I can automatically store these logs in a file? I probably would need them in future for auditing purposes.
Cheers,
Vishy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Yes. Do an all terms search for archive and log. This is one of the posts you will find.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.