Terrible Performance due to log file

kwwilliams · Post by **kwwilliams** » Mon Jan 16, 2006 9:38 am

jamesrender wrote:
chulett wrote:
I think that it looks like a really useful tool

It is a very useful tool, but you need training to be able to use it properly. Hopefully they will not pull the tool because they have not given proper training, that would be the result with any tool, not just DataStage.

jamesrender · Post by **jamesrender** » Mon Jan 16, 2006 9:48 am

kwwilliams wrote:
jamesrender wrote:
It is a very useful tool, but you need training to be able to use it properly. Hopefully they will not pull the tool because they have not given proper training, that would be the result with any tool, not just DataStage.

bit of a chicken and egg situation, I can't convince them to keep it without being proficient with it, and I can't get that proficiency if it goes in the next month.. their feeling is that its an extra level of complexity for the application, will probably want some sql/loader process created, even though that is what DS is all about. any opportunity to save a few bucks by dropping a license..

I'm sure that I'll discover a thorny issue that datastage solved once the decision to drop it has been taken..

chulett · Post by **chulett** » Mon Jan 16, 2006 10:25 am

kwwilliams wrote:
jamesrender wrote:
chulett wrote:
I think that it looks like a really useful tool
It is a very useful tool, but you need training to be able to use it properly. Hopefully they will not pull the tool because they have not given proper training, that would be the result with any tool, not just DataStage.

Be careful with your quoting... that wasn't me.

srinagesh · Post by **srinagesh** » Mon Jan 16, 2006 11:01 am

try altering the table by giving "nologging" option

Alter table Emp NoLogging;
(where emp is the name of the table)

This will definetly speed up the processing and will also avoid writing to the Oracle Logs.

HTH
Nagesh

jamesrender · Post by **jamesrender** » Tue Jan 17, 2006 2:46 am

Having just had a quick scan in my Oracle book, I don't believe that nologging will offer any benefit.

Viswanath · Post by **Viswanath** » Wed Jan 18, 2006 5:03 am

Hi,

Couldnt help asking. Ray you said that keepipng logs of DS for a long time without purge would degrade performance? We keep getting requets of data load dating back to more then 12 months, due to which we dont delete any logs. Would purging them help in performance improvement? Also is there a way where I can store these log files? Outside of DS I mean so that I can do some archiving?

Cheers,
Vishy

jamesrender · Post by **jamesrender** » Wed Jan 18, 2006 10:05 am

Viswanath wrote:Hi,

Couldnt help asking. Ray you said that keepipng logs of DS for a long time without purge would degrade performance? We keep getting requets of data load dating back to more then 12 months, due to which we dont delete any logs. Would purging them help in performance improvement? Also is there a way where I can store these log files? Outside of DS I mean so that I can do some archiving?

Cheers,
Vishy

I can't speak with any authority, but certainly in my case, the size of the log had a direct impact on job performance. It is possible to save DS logs to a file system. I have done it once, but I can't remember how.. sorry.

kcbland · Post by **kcbland** » Wed Jan 18, 2006 10:45 am

The logs degrade performance only when the job has to put a message into its log file. Since the log file is actually a dynamic hash file, an extremely large hash file may take longer to add a row, especially if the file just happens to need to dynamically expand at that point in time. If the job is logging a lot of messages, then yes, absolutely you will notice a severe impact to runtime performance.

At the start and end of processing, the job logs informational messages. There is an impact to total runtime, so keeping the logs purged allows startup and wrapup processing to be minimal. You will notice two reasons a job finishes moving data, yet still seems to be doing something else before finishing: logging/purging messages and updating the &PH& information. For faster startup/wrapup, purge your log files and keep the &PH& directory in the project clean.

Viswanath · Post by **Viswanath** » Wed Jan 18, 2006 9:33 pm

Thanks Kenneth.

I guess I have some work to do then. We have around 300 odd jobs in our Production project. I have to do a manual purge now and the enable auto purge for these.

But is there a way by which I can automatically store these logs in a file? I probably would need them in future for auditing purposes.

Cheers,
Vishy

ray.wurlod · Post by **ray.wurlod** » Wed Jan 18, 2006 10:55 pm

Yes. Do an all terms search for archive and log. This is one of the posts you will find.

Viswanath · Post by **Viswanath** » Wed Jan 18, 2006 10:57 pm

Thanks Ray.

Cheers,
Vishy