btprobe and re-indexing data

[mikehodges01]

I'm getting ready to finalize a Splunk install and roll it out for use... during my testing phase I added a bunch of data to my index that I don't need (eg, via syslog, WMI, legacy machines, etc). What I'd like to do, to conserve disk space and clean up my searches, is clean the entire index on my indexer and re-index everything from my universal forwarders (at this point only 5 or 6 machines). I've figured out that I can mount a remote filesystem on my indexer and use btprobe to look at the remote files, but I don't know what I need to put in for the -d argument to get the forwarder to send all its old data again. I've tried resetting the keys I get from the default folder that contains btree_index.dat, and I've tried doing it for the actual log file keys - but the indexer spits out a key and says "record not found". I know I could just remove the information from searches but I'd really like to be able to get the disk space back by starting fresh, and I'd like to know how to re-capture old log information should I ever need to in the future... it just seems like something kinda important in my environment. Thanks for any help you may be able to give me!

[bmacias84]

@mikehodges01,
I needed to perform task in re-sending data from the forward and reseting my test indexes. The easy part of clearing my index, but re-sending data from the forward took a little bit of work.

Clearing indexes:

splunk stop

splunk clean eventdata -index mytestindex -f

splunk start

This is the easy part.

Now to have the forwarder resend previous indexed data, requires the uses of the btprobe tool. To view all the entries you ./btprobe -d SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db -k ALL. If notice that all the values are in Hex and are displayed in key=0x<somevalue> sptr=<somevalue>.

To reset one file in your fishbucket from your UF:

./btprobe -d SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/access.log --reset 

#

Note: Your splunk daemon or service must be stopped and if you try to use the base directory you will recieve "could not open '/var/log/' (Access is denied.)".  If you doing this on a UF the btprobe will need to be copied to your UF. 

You should recieve the following message: key=0x7b100f52c71e67f1 scrc=0x6e16f6c553b73581 sptr=8070178 fcrc=0xcdcd2890474a56fb flen=0 mdtm=1347001048 wrtm=1352329761**95Record (key 0x7b100f52c71e67f1) reset.

If in my case I have hundreds of test file sitting on a UF in mulitple directories, so I wrote a perl script to feed each file in each directory to the btprobe utility.

NOTE: THIS WAS WRITTEN FOR 5.x, but still works in up to 6.5

Hope this helps you or gets you started. Dont forget to accept or thumbs up answers.

Cheers,

[yoshispendiff]

Running splunk cmd btprobe instead of just ./btprobe in 6.2.1 works.

[BobM]

Splunk tracks which file it has seen in it's thefishbucket index. On a heavyweight or lightweight forwarder, you can run the following ON THE FORWARDER to clean data that was previously collected.

splunk stop
splunk clean eventdata -f
splunk start

on a universal forwarder you need to delete the fishbucket folder.

delete $SPLUNK_HOME/var/lib/splunk/fishbucket
splunk restart

[mikehodges01]

That didn't seem to work for me. I ran

rmdir /s c:\program files\splunkuniversalforwarder\var\lib\splunk\fishbucket
splunk restart

on a universal forwarder and the old data was not re-indexed. Am I missing a step or something? However, I did manage to achieve my ends by uninstalling and reinstalling the universal forwarder. I suppose this solution will suffice for now but I'd still like to find a way to re-index anything I want without having to re-install the universal forwarder.