Detecting Unused Index and Reducing Splunk Log Size

[megha_04]

Is there a way to detect unused indexes in Splunk via a query? Also, how can we control the growth of log sizes effectively?

[livehybrid]

In terms of understanding which indexes are NOT being accessed. This is actually pretty challenging for a number of reaons, whilst its possible to look in the _audit index and see which indexes are being searched, its pretty difficult to determine exactly which indexes have been searched for a number of reasons:

• Different users have access to different indexers, so using wildcards (e.g. index=*) can mean different indexes are accessed depending on roles.
• Macros/tags/eventtypes may contain index references and would need to be determined and expanded
• Different user roles may have different srchIndexesDefault which means they might not specify an index to search as rely on the defaults.

Are you using Smartstore/Splunk Cloud? This may offer some slightly different approaches to this as we could look at smartstore cache activity to try and determine indexes accessed.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

Add to this the fact that searches can be created dynamically by means of subsearches and/or map command and there is no way to find all indexes (not) accessed by looking at searches.

One could hypotesize that you could try to leverage some OS-level monitoring to find whether the actual index directories are accessed but that could also not yield reasonable results since Splunk's housekeeping threads must access the indexes to enforce retention policies and data lifecycle.

Having said that - you can search _internal and _audit logs for executed searches and try to build a list of indexes which were used and thus limit your investigation whether anyone uses the ingested data to only a subset of indexes not mentioned in that list.

[livehybrid]

Hi @megha_04 

Regarding controlling the sizes of logs - I would recommend looking at https://www.splunk.com/en_us/blog/tips-and-tricks/managing-index-sizes-in-splunk.html as there is a little much to fit into an answer here!

But typically it is managed by setting the frozenTimePeriodInSecs per index to control how long (in seconds) your index retains data for.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing