Getting Data In

remove source and source types

dhs_harry08
Path Finder

Hi

I was testing splunk by adding multiple servers by both syslog and splunk forwarder. But now my splunk indexer has all kinds of stuff that I don't need anymore.

I want to remove some hosts, source and source types and the data it contains. Per the document I should be able to delete by removing stanzas in indexes.conf. But i can't find the entries in it.

Please help me out.

Regards,
Harish

Tags (3)
1 Solution

kristian_kolb
Ultra Champion

No,

to stop new events from coming into your indexes - remove the [monitor] and TCP/UDP stanzas in inputs.conf or go through the GUI to remove them. This, however does not remove any data already indexed.

If you delete stuff from indexes.conf, you remove the reference to where the indexes are located on disk. I don't think that's what you want.

If you selectively want to remove some of the information in the indexes you otherwise wish to keep - then you should type in a search that selects those events and pipe them to delete.

Make sure that your user role has the can_delete privilege (Manager -> Access Control -> Roles -> your_role ). Then, assuming you have data from one host (badhost) you wish to delete and data from another host (goodhost) you wish to keep:

host=badhost | delete

Whoops, now all data from badhost is gone. However, the data from goodhost is of two different sourcetypes (badtype and goodtype) - guess what:

host=goodhost sourcetype=badtype | delete

..and it's gone as well.

You should note that this operation does NOT actually delete the data, it just prevents it from ever being returned as search results.

If you want to clean an index completely, then you should do so through the command line:

  • stop splunk

  • type /opt/splunk/bin/splunk clean eventdata -index <the index you wish to clean>

  • start splunk again

For more information regarding the clean command see:
http://splunk-base.splunk.com/answers/8191/is-there-a-way-to-clean-event-data-from-a-specific-date-r...

hope this helps,

Kristian

EDIT: Missed a parameter in the clean command... oops. fixed now.

View solution in original post

kristian_kolb
Ultra Champion

No,

to stop new events from coming into your indexes - remove the [monitor] and TCP/UDP stanzas in inputs.conf or go through the GUI to remove them. This, however does not remove any data already indexed.

If you delete stuff from indexes.conf, you remove the reference to where the indexes are located on disk. I don't think that's what you want.

If you selectively want to remove some of the information in the indexes you otherwise wish to keep - then you should type in a search that selects those events and pipe them to delete.

Make sure that your user role has the can_delete privilege (Manager -> Access Control -> Roles -> your_role ). Then, assuming you have data from one host (badhost) you wish to delete and data from another host (goodhost) you wish to keep:

host=badhost | delete

Whoops, now all data from badhost is gone. However, the data from goodhost is of two different sourcetypes (badtype and goodtype) - guess what:

host=goodhost sourcetype=badtype | delete

..and it's gone as well.

You should note that this operation does NOT actually delete the data, it just prevents it from ever being returned as search results.

If you want to clean an index completely, then you should do so through the command line:

  • stop splunk

  • type /opt/splunk/bin/splunk clean eventdata -index <the index you wish to clean>

  • start splunk again

For more information regarding the clean command see:
http://splunk-base.splunk.com/answers/8191/is-there-a-way-to-clean-event-data-from-a-specific-date-r...

hope this helps,

Kristian

EDIT: Missed a parameter in the clean command... oops. fixed now.

manidandu
Explorer

I want to delete data for a source as we are seeing sensitive information in it. Can I use | delete for that source? Once we fix the maxing issue, we can start pulling logs from that source and we can search it right? Please let me know.

0 Karma

ranjyotiprakash
Communicator

Thanks a lot kristian.kolb !!
this answer helped me a lot ....

0 Karma

rupesh_patil20
Path Finder

What success message does it gives when we delete the index or source?

0 Karma

koredyte
Engager

Splunk -> Manager -> Data inputs -> (Files and Directories in my own case) -> locate your data source and on the right side under "Actions", click delete.

Hope this helps

cvajs
Contributor

what i notice in v4.3.1 for linux is that source types are created in weird ways. if you create a new source (file) and you dont preview the data and you use manual source name, the source is not added to the props.conf file and will never be seen in the "list" when creating a new source, but, if you use the same name again it will complain with a "that name already exists" error message. but if you preview the data and select "create new source" it will populate props.conf with the new source type and will be available in the "list". but no place to edit/delete sources from the GUI?

stefanlasiewski
Contributor

Certain types of sources do not appear in this list.

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...