Often times, we are tasked with deleting data out of an index to trim it down. Generally, we do this by setting the frozenTimePeriodInSecs and allowing Splunk to remove old data.
However, before we do that, we need to figure out where the best bang for the buck is. Many times people just ask us for the oldest event and try to plan it around there, but this is usually a very small bit of the data ( some old remnants of events that somehow got indexed way later ). In order to figure out what the earliest time is that there is a large chunk of data at, we've been doing something
index=myindex | timechart count span=1w
And running that across All Time. The problem is that it's slow, especially for our bigger customers, who are often in the 1 billion event range for a few months of data. Plus, we're just doing event counts, rather than getting real sizes of the data ( or even the compressed on disk size ). I've played with using len(_raw) and summing it up, but it just made things even slower.
Is there a quicker way to do this? I've played with dbinspect but that doesn't give the breakdown we'd need. I've also thought about doing this as an ongoing search with a scripted input that runs daily to get the index sizes on disk, but I realized that this doesn't give us a picture of what Splunk looks right now. The second we delete any data, all that historical data becomes obsolete. Ideally, I'd like to get a current size of index, per index/per day.
Anyone have any ideas?
... View more