Impact of DS logs

[vputta]

HI - We are thinking to not automatically purge any of the DS logs in our project. With this option all the logs will be piled up and we will have a chance to see all the logs for all the years. Will this consume more space and will it impact performance of any of DS jobs? Plz suggest.

Thanks

[jwiles]

That is a bad thought and I would not recommend it. Rather, if you're required to keep the logs, institute a practice of regularly extracting the job logs to text files and storing them in another location. As the number of logs stored (and hence the size of the log hash files) grows, you will find that Director performance (and ultimately job performance) will suffer. Depending upon configuration options, you will hit the limit of log file size (2GB for 32-bit files) which will cause other problems (failed jobs is one result).

"Piled up" is a good metaphor here, in the negative sense...if it's piled up you will have problems.

Regards,

[vputta]

Thanks for the quick response. Can we go with an option of 6 months?

[jwiles]

I can't tell you that...it will depend upon how heavily your system is used. In most environments, you should be thinking in terms of days or a few weeks at most, 6 months is a LONG time to keep logs available for viewing in Director. That's the reason for the suggestion to archive the logs elsewhere, so that they are easily available for viewing and backup purposes.

Regards,

[ray.wurlod]

You could change particular logs to 64-bit addressing, which would give a much larger theoretical size. But is your disk big enough for that? If your disk fills you get even worse (and unrecoverable) problems.

Do archive your logs regularly, perhaps even more frequently than you purge them, if you need to be able to guarantee that the information contained in them is preserved.

[PaulVL]

More than 2 days of logs already impact 1 of my projects. I can't even imagine 6 months or a year.

Extract them once per day, FTP them to a repository.

Remember to implement a purge process on your repository and nuke anything older than X months.