Hi everyone,
I received a disk usage alert from my service tier server this morning. Upon further review of the disk I am seeing that the IBM\InformationServer\shared-open-source\kafka\logs directory is 12GB. It does not appear that any of the logs are bloated per se, just a steady accumulation of log files. Does anyone know how to address this? The logs go back for a couple years, I assume there must be a log retention setting that can be configured?
Kafka logs on Service tier
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 145
- Joined: Thu Oct 26, 2017 10:43 am
-
- Premium Member
- Posts: 425
- Joined: Sat Nov 19, 2005 9:26 am
- Location: New York City
- Contact:
Two ways come to my mind:
1) change the retention period for events on the topic
$InstallPath/shared-open-source/kafka$ update-config.sh --zookeeper serverName:port --entity-type topics --alter --add-config retention-ms=500 --entity-name InfosphereEvents
2) delete the topic. It will be automatically recreated
1) change the retention period for events on the topic
$InstallPath/shared-open-source/kafka$ update-config.sh --zookeeper serverName:port --entity-type topics --alter --add-config retention-ms=500 --entity-name InfosphereEvents
2) delete the topic. It will be automatically recreated
Julio Rodriguez
ETL Developer by choice
"Sure we have lots of reasons for being rude - But no excuses
ETL Developer by choice
"Sure we have lots of reasons for being rude - But no excuses
-
- Premium Member
- Posts: 145
- Joined: Thu Oct 26, 2017 10:43 am
Thanks for the reply! I have been reading a bit about altering the log retention settings. I see you're calling the update-config script and passing in new retention attributes from there. Is this different than editing the server.properties file itself in $InstallPath\shared-open-source\kafka\conf ?
I'm on a windows server unfortunately. I'm also not entirely sure I understand what a topic is in this context.
I'm on a windows server unfortunately. I'm also not entirely sure I understand what a topic is in this context.
-Me
-
- Premium Member
- Posts: 425
- Joined: Sat Nov 19, 2005 9:26 am
- Location: New York City
- Contact:
It should be ok to add the new parameter to the configuration manually. My preference is to use the commands provided to do all the maintenance tasks
A Topic is just a stream of records, In the Kafka or streaming world is called topic. For Information Server the default name of the topic is InfosphereEvents
A Topic is just a stream of records, In the Kafka or streaming world is called topic. For Information Server the default name of the topic is InfosphereEvents
Julio Rodriguez
ETL Developer by choice
"Sure we have lots of reasons for being rude - But no excuses
ETL Developer by choice
"Sure we have lots of reasons for being rude - But no excuses
-
- Premium Member
- Posts: 145
- Joined: Thu Oct 26, 2017 10:43 am
Official word thus far from IBM is to delete the log files. I did that back to January of this year for the kafka-server log files only. I am awaiting a response from them on proper configuration of server log file retention as it was indicated that those logs are not affected by the config file.
-Me
-
- Premium Member
- Posts: 145
- Joined: Thu Oct 26, 2017 10:43 am
Official word back from IBM was that log retention configuration for kafka-server.log files cannot be done via any log files. These need to be manually cleaned or scripted to maintain a limited retention. I created a scheduled task that runs the following command:
Replace 'D:\IBM\InformationServer\shared-open-source\kafka\logs' with your IBM InformationServer directory for your logs.
Code: Select all
cmd /c powershell -command "Get-ChildItem 'D:\IBM\InformationServer\shared-open-source\kafka\logs' -Recurse -File | Where { -not $_.PSIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-60) -and $_.Name -like 'kafka-server.log*'} | Remove-Item -force" && exit
-Me