Getting Data In

removing data of a particular sourcetype in an index

joydeep741
Path Finder

How to remove data of a particular sourcetype in an index ?

index=myindex has three soucetypes , st1, st2 and st3.

I want to remove all data of st2 . How to do that ?

Tags (2)
0 Karma
1 Solution

renjith_nair
SplunkTrust
SplunkTrust

You can delete it by

index=indexname sourcetype=st2|delete

This will not delete the data from disk but just mark it as deleted so that it won't be visible in searches. Once the index retention period is reached, this will be deleted permanently.

Reference :
http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/RemovedatafromSplunk#Delete_events_from_su...
https://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Delete

View solution in original post

di2esysadmin
Path Finder

We are over our indexing limit - can deleting data get us back under that limit?

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

No, you can't do that. Licensing meter is measuring the data while you inject and it does not really care what you are doing with the data after that. So deleting data will not have any effect on indexing but soft delete (with the delete command) helps you to hide some data and hard limit (deleting from disk by setting retention policy) will help you to save some disk space.

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

You can delete it by

index=indexname sourcetype=st2|delete

This will not delete the data from disk but just mark it as deleted so that it won't be visible in searches. Once the index retention period is reached, this will be deleted permanently.

Reference :
http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/RemovedatafromSplunk#Delete_events_from_su...
https://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Delete

View solution in original post

AnilPujar
Path Finder

index=indexname host=server1 | delete

I have done the same way, because i don't want those data to come up in search. Since i have huge hard disk capacity so no tension with junk unwanted data which i filter out search query itself.

My Splunk Architecture, 4 indexer in cluster.

Now the problem is, when I push some new configuration to indexers through cluster master, sometimes it does rolling restart, and when rolling restart happens, the above deleted events will again show up in search results. May be due to replication factor it again shows up.

Please suggest how to avoid it from showing it again.

0 Karma

AnilPujar
Path Finder

Important fields which need to be retained: _time, sourcetype
Few steps which I followed but couldn't succeed.

  1. Using Summary Index Has a Backup index.

index="main_primary_index" "search filter terms for specific data"
| table _time , required_field1,required_field2,required_field3...
| collect index=bkp_index sourcetype="required_st_name"

Problem:

_time is taking current system time, tried all the possible scenarios like strptime, strftime, convert ctime

in collect command there is no arguments to retain _time as it is.

0 Karma

AnilPujar
Path Finder
  1. Thought to export the required data in csv file and upload it again after cleaning the complete index.

if I use default sourcetype csv while uploading the file, the _time is syncing perfectly fine with my _time in csv file.

But I need to retain my souretype name.

so I cloned csv sourcetype with required sourcetype name and then when uploading the csv file back to cleaned index the _time is taking current system time.

0 Karma

AnilPujar
Path Finder

@TStrauch , @renjith.nair for me _time and sourcetype is crucial so how to retain and index back the data. please see my above comments.

0 Karma

TStrauch
Communicator

Hi,

first: Its not possible to delete specific events from your index and get the disk space back. You need to clean the whole index to do that.

second: There is an role in splunk which gives you the option to delete events based on a specific search. This command will not delete the events from you disk. It will only eliminate them in search.

Search for "can_delete" on splunk docs. The role is not assigned to anyone by default. Even the admin don´t have it.

!!!BUT: Be carefull by execute the "delete" command. You cannot make it undone!!!

Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.