topic Re: no longer seeing all logs after clearing index in Splunk Search

no longer seeing all logs after clearing index

jonathanfalconi — Fri, 23 Nov 2012 08:44:10 GMT

I am no longer seeing all my logs on the indexer after clearing the index of all data. Is there something that needs to be cleared or restarted on the forwarder so all the available logs can be gobbled up again?

I used this command on the indexer

splunk clean eventdata -index proxylogs -f

Re: no longer seeing all logs after clearing index

Drainy — Fri, 23 Nov 2012 08:52:16 GMT

What command did you use to clear the indexes? Splunk stores a record of what it has read in something called the fishbucket, these exist on forwarders too so you need to clear them on an indexer (if its reading local files) or a forwarder (if its reading local files on a remote server)

From memory I believe the command is;

./splunk clean eventdata _fishbucket

If you don't have anything else of importance in other indexes or want to do this on a forwarder then you can just do a clean all.

Re: no longer seeing all logs after clearing index

Drainy — Fri, 23 Nov 2012 09:04:08 GMT

Right, so you need to clear the fishbucket, only a clean all would hit the fishbucket too

Re: no longer seeing all logs after clearing index

jonathanfalconi — Fri, 23 Nov 2012 09:22:57 GMT

I am running version 4.2.4 on Solaris - when I run the command I get he following error:

This action will permanently erase all events from the index '_fishbucket'; it cannot be undone.
Are you sure you want to continue [y/n]? y

ERROR: Cleaning eventdata is not supported on this version.

thanks
Jon

Re: no longer seeing all logs after clearing index

Drainy — Fri, 23 Nov 2012 09:35:00 GMT

hmm, perhaps ./splunk clean eventdata -index _fishbucket or if not, is there other data you need or could you reindex it all? (depends if this is prod or not really..) You could just run ./splunk clean eventdata (Warning, this deletes everything)

Re: no longer seeing all logs after clearing index

jonathanfalconi — Fri, 23 Nov 2012 09:48:55 GMT

strange - seems I have another issue now! I have tried all the variations of the command and still cannot clean the index always comes back with this error: ERROR: Cleaning eventdata is not supported on this version.

./splunk clean eventdata -index _fishbucket
./splunk clean eventdata

Re: no longer seeing all logs after clearing index

Drainy — Fri, 23 Nov 2012 10:21:29 GMT

try running ./splunk help clean and see what it says 🙂 The docs seem to match what I've pasted but its clearly not happy.

Re: no longer seeing all logs after clearing index

jonathanfalconi — Fri, 23 Nov 2012 10:37:41 GMT

Looks like this command is no longer supported, I have seen one other person with same issue but no solution. Will start new thread "clean eventdata command not supported on UF"

Thanks for your help.

Re: no longer seeing all logs after clearing index

Drainy — Fri, 23 Nov 2012 10:47:17 GMT

Ah, in that case you will just need to delete the fishbucket manually, use an rm -rf on the var/lib/splunk/fishbucket directory within the forwarder directory. Make a backup first but this should do the job

Re: no longer seeing all logs after clearing index

jonathanfalconi — Fri, 23 Nov 2012 13:23:55 GMT

I have cleared out the index on my indexer and the fishbucket on the Universal forwarder but I am still only receiving logs from one particular file in the directory being monitored, the directory has multiple files which should be feeding into the indexer. The tailing message I get is below: INFO TailingProcessor - Archive file

11-23-2012 13:20:22.718 +0000 INFO TailingProcessor - Archive file='/var/opt/proxy/logs/lxnhostp01/access1211231318-x.2x3.1x4.x.log.gz' has stopped changing, will read it now.

Any thoughts?

thanks
Jon

Re: no longer seeing all logs after clearing index

jonathanfalconi — Fri, 23 Nov 2012 13:27:02 GMT

had to manually clear out the fishbucket rm -rf all files in fishbucket due to the following error:

"strange - seems I have another issue now! I have tried all the variations of the command and still cannot clean the index always comes back with this error: ERROR: Cleaning eventdata is not supported on this version.
./splunk clean eventdata -index _fishbucket
./splunk clean eventdata"