When splunk is ingesting data we get a warning about 10% FS Availability

[mark71]

Hi. I am a new splunk user with a question:

When splunk is ingesting data we get a monitoring system warning about 10% FS Availability. Then the FS space returns to a value > 10% availability. Is there a file/location where temporary data is written while ingestion is happening?

Thanks 

[_JP]

What type of Splunk instance are you getting this notification on - your Indexer, or are you using a single Splunk instance? I'm also assuming this notification is coming from outside of Splunk - like an OS monitor or something like that.

For example, an Indexer is generally going to put lots of stuff into $SPLUNK_HOME/var/lib/splunk because that's where the indexes are stored.  

The first thing I could think of is your indexer drive is almost full, and you index more data causing you to dip below a Splunk-configured max index size.  This causes Splunk to then move/delete old buckets of data because it dipped below the configured threshold.  

Here's docs on this this concept:  Configure maximum index size - Splunk Documentation

Also, these configs would be in one of your indexes.conf files.

[mark71]

Hi _JP

Thanks for the response. Yes, the instance is an indexer.  I have read the linked documents and I understand more of the detail about the indexer and how it stores various stages of data. I'll review and in the meantime we will be adding additional FS space.

Thank You