Getting Data In

Delete the data after indexing

abhayneilam
Contributor

lets say daily I recieve 5 files, and I am indexing 5 files and running my query to generate the report. Now, my requirement is after running the query , data from the index should be deleted, Next day I will put another 5 files in the same index and run the query to generate the report.

Kindly help me in this regard.

Thanks in Advance,
Abhay

1 Solution

bmacias84
Champion

There are two ways I know. The first is using clean from splunk cli and the delete command within search.

The clean command cleans (deletes) all eventdata from the index. The catch is you must stop your splunk indexer and then restart after running the command.


$SPLUNKHOME/bin/splunk stop
$SPLUNKHOME/bin/splunk clean eventdata -index myindex -f
$SPLUNKHOME/bin/splunk start

The search delete command has to be granted expicity to a user or role, even the admin does have the ability to delete unless specified. All the delete command is a sudo delete it only flags the events as unsearchable, giving the appearance of being deleted. So you will not reclaim disk space.


source="myfiles*.log" | delete

View solution in original post

jumah35
New Member

Thank you guys for your replies...
I am currently out of office. I will check in a week and keep you posted.
Regards
Julien

0 Karma

jumah35
New Member

Oups !! sorry ...
Julien

0 Karma

MuS
SplunkTrust
SplunkTrust

wow - that's a long out of office .... 🙂
could it be, that you used the wrong answer here? the last activity on this was almost exactly two years ago 😉

0 Karma

kristian_kolb
Ultra Champion

Or for a more automatic approach that involves a little more initial configuring, you can set retention parameters indexes.conf

This can be a bit more tricky, since the rather short retention requirements you have are a bit unusual.

I believe (haven't tried it myself, at least not all of them in this type of combination) that you for your index can set the following parameters;

[your_index]
maxTotalDataSizeMB = XXXX
maxDataSize = XXXX
maxHotBuckets = 1
maxHotIdleSecs = 28800
maxWarmDBCount = 0
coldPath.maxDataSizeMB = 1

where XXX should be something like 10 times more than you would expect the size of your 5 files to be, combined. This is in MB. So if you have 5 files, each about 50 MB in size, set this value to 2500.

In essence what this does (should do) is to take all the incoming events and store them in a hot bucket. After 8 hours (28800 seconds) the hot bucket is moved to a warm bucket. But since the maximum number of allowed warm buckets is 0, Splunk should move it straight to a cold bucket. If (and this is another assumption) your indexed data exceeds 1 MB, it will be frozen (i.e. deleted unless you've explicitly configured a coldToFrozenDir or coldToFrozenScript)

Please note that you should not store any information you wish to keep in this index, or to store more information than you have specified. It will just get deleted.

More information can be found here;
http://docs.splunk.com/Documentation/Splunk/5.0/admin/AboutConfigurationFiles

http://docs.splunk.com/Documentation/Splunk/5.0/Indexer/HowSplunkstoresindexes

http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Wheretofindtheconfigurationfiles

http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Indexesconf

If you decide to try this out, please report back on the outcome. You're the guinea pig in this experiment 🙂

Godspeed,

Kristian

bmacias84
Champion

There are two ways I know. The first is using clean from splunk cli and the delete command within search.

The clean command cleans (deletes) all eventdata from the index. The catch is you must stop your splunk indexer and then restart after running the command.


$SPLUNKHOME/bin/splunk stop
$SPLUNKHOME/bin/splunk clean eventdata -index myindex -f
$SPLUNKHOME/bin/splunk start

The search delete command has to be granted expicity to a user or role, even the admin does have the ability to delete unless specified. All the delete command is a sudo delete it only flags the events as unsearchable, giving the appearance of being deleted. So you will not reclaim disk space.


source="myfiles*.log" | delete

bmacias84
Champion

This has to be completed on each Indexer if your are running a distributed search configuration. Also you can clean with windows admin accoutn and splunk user that has the admin role? Did you specify index or use "clean all -f".

0 Karma

gjohnson
New Member

Didn't seem to work. I stopped both splunkd and splunkweb (windows implementation) and issued the 'splunk clean eventdata' command (as admin) - it warned that all data would be lost. I agreed. The CLI indicated that all databases had been cleaned except:
Disabled database 'splunklogger': will not clean
When I restarted splunk services all of the index data was still there and all the settings were still intact.

0 Karma

bmacias84
Champion

Once an index is cleaned the data is gone (all eventdata in that index), there is no going back. If you have a distributed search with muliptle indexer and indexes you will have to perform this action on all indexers.

0 Karma

abhayneilam
Contributor

How to stop "splunk indexer" ? can you please tell me in steps how to "clean the event"

Thanks in advance !!

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...