To identify unused/unsearches data in Splunk

[rahulhoney]

Is there a way to find unused/unsearched data in Splunk?

Example:
In an Index=XYZ we are ingesting 100GB of data on a daily basis.

Out of that 100 GB when we run queries we are retrieving 60GB of logs and the remaining 40GB never retrieved or never searched upon.
And using this scenario we can send those events to the NULL queue.

[somesoni2]

I don't think there is an easy way to do that. It would be easier if you could talk to the consumer of the data (Splunk users who runs those queries) and find out what type of data they are interested in and what's ok to drop. E.g. there could be some healthcheck type of INFO events that they might not use, so you could drop those.

[rahulhoney]

Thanks, @somesoni2 but we are trying to figure it out without asking consumer and sender too.

[somesoni2]

There is no way in Splunk to track which specific data is being used. Again, data that is not queried till today, doesn't mean that it won't be queried tomorrow (or even 5 mins from now), so it's a risky business removing/filtering data. Your consumers and senders could only give you 100% accurate details about this.

Other thing you could do (again this is not 100% accurate) is to look at current adhoc and scheduled searches running in your environment, look at their search string to find which sourcetype/sources from that indexes are being used. Other sourcetypes can be a candidate for removal.