Splunk Search
Highlighted

Is there a search to delete logs completely from an indexer after 3 months in an indexer clustering environment?

Explorer

I have an indexer cluster environment and need to delete the logs completely from the indexer:

source=* sourcetype=* host=* latest=-90d@d earliest=0

I have the above search where the I can pipe it with |delete or | splunk clean eventdata, but found that piping to delete does not reclaim disk space and the clean command does not work on clustered indexes.

Is there any other search to do it, or do I have to retain it on the bucket until rolling to frozen?

0 Karma
Highlighted

Re: Is there a search to delete logs completely from an indexer after 3 months in an indexer clustering environment?

SplunkTrust
SplunkTrust

There is no search that can do this: piping to delete simply makes results not show up in searches, it does nothing to actually "remove" the results from disk.

There is an approximate answer (within a few days) to your problem though, but I believe there's no hard and fast way to get it precise. But as long as "pretty close" works well enough (i.e. the problem is just disk space, not some insane non-auditability requirement that has a hard and precise 3 month limit), then the solution is to control your buckets.

A good reference to read first is How Splunk Stores Indexes. Near the bottom, they link to another do on setting retirement and archive policy. In there, check the section on "Freeze data when it grows too old: Set frozenTimePeriodInSecs". The deal here is that you can set a time in seconds (make sure you add a bit to account for leap years and different sized months and all that - hence why I mention there's no "perfect" answer to your problem, only a an approximate one) to roll your buckets to frozen and so as long as you have no definition on what to do with frozen buckets, the default is to delete them.

You have a complicating factor of bucket sizes, too - buckets won't be frozen/deleted until ALL the events in them are older than your setting. So, if you buckets span 5 days, you'll have data available for about 3 months plus 5 days (not bad), but if they span a month then you'll have 4 months on disk usually. Or, well, you'll have UP TO 4 months on disk, then it'll delete old buckets and you'll be back down to three. I'd not worry about this until after you've tried the above and only then if you find the results not good enough, but if you do find that's the case, look up maxDataSize in indexes.conf. You can adjust that a bit to get somewhat smaller buckets (I wouldn't go overboard) if you need to. The S.o.S. app (and Fire Brigade app, and the new DMC in 6.3ish) all should have some bucket info for you if you want to see that.

0 Karma