topic Re: not indexing after cleaning eventdata in Getting Data In

not indexing after cleaning eventdata

hiddenkirby — Wed, 18 Aug 2010 23:55:42 GMT

So i created an app folder... and indexes.conf .. and an inputs.conf to monitor a directory.

I then restarted splunk via CLI and everything was perfect. Lots of data being indexed immediately.

I realized my sourcetype was wrong, so i...

stopped splunk
made my change to the inputs.conf file for the name of the sourcetype i wanted
did a splunk clean eventdata
started splunk

[monitor:///data/splunk/mydata/] index = mydata sourcetype = mysourcetype crcSalt = disabled = false

i have 0 events, and can't seem to populated the index again.

does this have to do with the "crcsalt = " line i have in my inputs.conf?

i have even, created a new different index ... still no go.

thoughts?

Re: not indexing after cleaning eventdata

Stephen_Sorkin — Thu, 19 Aug 2010 00:03:44 GMT

Did you clean all indexes or just a single one with splunk clean eventdata? If you cleared all, splunk should have removed the contents of $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db/ which contains the position in files being read. If you just cleaned a single index, that would not have been cleared and we will continue reading from the end of the files.

This shouldn't have anything to do with the crcSalt.

Re: not indexing after cleaning eventdata

hiddenkirby — Thu, 19 Aug 2010 00:06:46 GMT

well i don't want to clean them all. How does one clean one index and remove those entries from "fishbucket"?

Re: not indexing after cleaning eventdata

Stephen_Sorkin — Thu, 19 Aug 2010 00:19:48 GMT

That's trickier and unfortunately we don't have a published tool to modify our database of seek pointers. You can convince Splunk to reindex these files by adding them from the CLI using one-shot input that doesn't check the db. For each file you should run: splunk add oneshot /data/splunk/mydata/ -index mydata -sourcetype mysourcetype.

Re: not indexing after cleaning eventdata

hiddenkirby — Thu, 19 Aug 2010 01:01:57 GMT

This command does not work with gzips?

Re: not indexing after cleaning eventdata

Stephen_Sorkin — Thu, 19 Aug 2010 01:11:32 GMT

Should work fine with gzips, tars and zips. It doesn't work for directories.

Re: not indexing after cleaning eventdata

hiddenkirby — Thu, 19 Aug 2010 01:13:09 GMT

Also to confirm if i do ~/bin/splunk clean eventdata without naming an index it will clean them all .. including the fishbucket pointers? or do i need to clean "globaldata" or "all" to get that accomplished?

Re: not indexing after cleaning eventdata

hiddenkirby — Thu, 19 Aug 2010 01:14:10 GMT

i get "In handler 'oneshotinput': invalid file: path='/data/splunk/mydata/txt123.gz' error='Not a regular file'"

Re: not indexing after cleaning eventdata

Stephen_Sorkin — Thu, 19 Aug 2010 03:02:28 GMT

Yes, clean eventtdata without an index specified cleans all indexes, including the fishbucket pointers.

As for the "Not a regular file" error, that's unrelated to the gzipped nature, it means that the file doesn't show up to "fstat" as a regular file. Could you run "ls -l" on that file?

Re: not indexing after cleaning eventdata

BunnyHop — Thu, 19 Aug 2010 04:17:58 GMT

Maybe try and re-add the monitor? Remove the monitored directory and then re-add it.

Re: not indexing after cleaning eventdata

hiddenkirby — Thu, 19 Aug 2010 20:22:16 GMT

ahhah! apologies... and thank you.

Re: not indexing after cleaning eventdata

Wushu — Thu, 28 Jul 2011 12:05:38 GMT

I've had this problem when clearing eventdata - even if i included 'all' - I simply put crcSalt = mysourcetype in the inputs.conf and it fixed the problem.