topic Re: Modular inputs for Microsoft cloud services Add-on fill up the disk space??? in All Apps and Add-ons

Modular inputs for Microsoft cloud services Add-on fill up the disk space???

captcha — Tue, 05 Nov 2019 22:59:10 GMT

I am using Microsoft Cloud services Add-on to ingest the logs from Azure storage account. The modular inputs for the app does not have clean up mechanism, filling up the disk space and causing service interruptions. Are others using this app see the same issue? Do you have a batch jobs to clean up the files?

Re: Modular inputs for Microsoft cloud services Add-on fill up the disk space???

MuS — Wed, 06 Nov 2019 01:04:05 GMT

Hi captcha,

you can create a $SPLUNK_HOME/etc/log-local.cfg and add a log rotation configuration for that log file. Use the $SPLUNK_HOME/etc/log.cfg file as example. Find some basic information about the log management process here https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/WhatSplunklogsaboutitself#The_log_management_process

Hope that helps ...

cheers, MuS

Re: Modular inputs for Microsoft cloud services Add-on fill up the disk space???

MuS — Wed, 06 Nov 2019 01:07:31 GMT

In the log.cfg search for the appender.xyz.maxFileSize or appender.xyz.maxBackupIndex options for some examples.

cheers, MuS

Re: Modular inputs for Microsoft cloud services Add-on fill up the disk space???

hkubavat_splunk — Wed, 06 Nov 2019 04:59:28 GMT

I think Captcha is facing issue of Disk space because of files that is created from modular inputs and not the log files of modular input. Captcha can you please confirm?

Re: Modular inputs for Microsoft cloud services Add-on fill up the disk space???

captcha — Wed, 06 Nov 2019 18:09:00 GMT

Hello MuS and hkubavat,

I think I did not state my questions correctly. Looks like the modular inputs creates a checkpoint file, this file grows really big causing some disk space issues. Deleting these checkpoint files will initiate reindexing of the logs. Any suggestions on how to deal with this issue?

Re: Modular inputs for Microsoft cloud services Add-on fill up the disk space???

MuS — Thu, 07 Nov 2019 01:58:17 GMT

Okay to me that sounds like a bug in the mscs_checkpointer.py script, because you should only have the latest checkpoint in that file and not a list of checkpoints. Haven't checked very deeply but it sounds like it does an append instead of a replace in the checkpoint file.

Hope this helps ...

cheers, MuS

Re: Modular inputs for Microsoft cloud services Add-on fill up the disk space???

hkubavat_splunk — Thu, 07 Nov 2019 05:05:56 GMT

There is a limitation in API of azure where you cannot filter the logs as because of that you have to check all the files. In MSCS for one blob, there is one checkpoint file. So my question to Captcha is you are facing an issue is because of disk space or IO Read/write operation? It may be a case your CPU usage is normal though your system is slow.

Re: Modular inputs for Microsoft cloud services Add-on fill up the disk space???

francoisternois — Fri, 18 Mar 2022 12:31:54 GMT

Hi,

I'm facing the same issue. This folder (splunk/var/lib/splunk/modinputs/mscs_storage_blob) keeps growing with checkpoint files, especially on few storage account folders. Each file referencing the name of the storage account file that Splunk ingest.

I guess that it check all the files to compare with the files on the storage account (explaining performance issue) trying to find a new file to ingest.

Did you find a way to solve this issue ?

Re: Modular inputs for Microsoft cloud services Add-on fill up the disk space???

Rhidian — Thu, 04 Apr 2024 11:02:25 GMT

Did you ever resolve this issue?