How can I delete the data, which is coming from SH and the UF via the host field when there shouldn't be any files to monitor?

[tkwaller_2]

Hello

I have to be doing something incorrectly. I have an indexes app that stores our index configs. Small environment, 2 indexers 1 Search Head(SH) 1Universal Forwarder(UF). I added some data via the UI on the SH to an index named dev_tsv. Now I'm adding data to that index via a UF but am deleting the index to clear out the previous data.

Deleted the config from indexes.conf and restarted the indexers. Also removed any index config for this index from the SH, just to make sure. On the UF I removed the files from the monitored path. I restarted all hosts after that.

I added the index config back to 1 indexer and restart and all the data that was previously in the index is in search again. The logs in Splunk say they come from the SH and the UF via the host field but there should be no files there for them to monitor and I can confirm that via filepaths.

How can I remove this data? How is this data still there? Where is this data coming from?

[adonio · Accepted solution]

hello there

to completely clean an index, i would recommend the following:
remove all inputs stanzas (including monitor) or comment out
stop the indexers, run on each indexer the following command from bin directory
splunk clean eventdata -index dev_tsv (or other index name)
restart the indexers. validate the index exists and there is no data in it.
as for why it happen, try to search by the field _indextime (the time where events hit the disk in the indexer) and verify whether these are new events or old events.
read here for more relevant information:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/RemovedatafromSplunk#Remove_data_from_one_...

hope it helps

[tkwaller_2]

That worked
Never had this issue. Whenever the index needed to be deleted it always worked by removing the index stanza and restarting, will do this from now on.

Thanks
Todd