I am using Microsoft Cloud services Add-on to ingest the logs from Azure storage account. The modular inputs for the app does not have clean up mechanism, filling up the disk space and causing service interruptions. Are others using this app see the same issue? Do you have a batch jobs to clean up the files?
you can create a
$SPLUNK_HOME/etc/log-local.cfg and add a log rotation configuration for that log file. Use the
$SPLUNK_HOME/etc/log.cfg file as example. Find some basic information about the log management process here https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/WhatSplunklogsaboutitself#The_lo...
Hope that helps ...
log.cfg search for the
appender.xyz.maxBackupIndex options for some examples.
I think Captcha is facing issue of Disk space because of files that is created from modular inputs and not the log files of modular input. Captcha can you please confirm?
Hello MuS and hkubavat,
I think I did not state my questions correctly. Looks like the modular inputs creates a checkpoint file, this file grows really big causing some disk space issues. Deleting these checkpoint files will initiate reindexing of the logs. Any suggestions on how to deal with this issue?
Okay to me that sounds like a bug in the
mscs_checkpointer.py script, because you should only have the latest checkpoint in that file and not a list of checkpoints. Haven't checked very deeply but it sounds like it does an append instead of a replace in the checkpoint file.
Hope this helps ...
There is a limitation in API of azure where you cannot filter the logs as because of that you have to check all the files. In MSCS for one blob, there is one checkpoint file. So my question to Captcha is you are facing an issue is because of disk space or IO Read/write operation? It may be a case your CPU usage is normal though your system is slow.