We are about to transition from using Windows servers to run Splunk to using Linux servers. On the day we make the switch, I want to "reset" our Universal Forwarders that reside on a number of other servers to force them to resend all the data from the one file they monitor (Apache access log file) to the new Linux indexer so that it has a complete day's worth of data.
Is that just a matter of stopping the UF on each server, removing all files/directories under $SPLUNK_HOME/var/lib/splunk/fishbucket, and restarting the UF, or is there more to it?
For the record: the forwarder apparently keeps track of what it has sent to the indexer. Due to the Windows server crashing yesterday morning, I had to scramble and cut over to the Linux servers yesterday and it only indexed new events, it did not reindex any of the data that had been indexed on the Windows server before I switched all the forwarders to point to the Linux server (even though the file being monitored had data going back to midnight).
Does anyone know what file(s) I should have modified on the forwarder that would have forced it to reindex all of the data? I thought there should have been something in /var/lib, but I couldn't find it.
That is an option. Depending on how your UFs are managed and how many you have, another option may be to follow instructions to move your existing indexes from your old server to your new server (setting up indexAndForward from the old server to the new server) http://wiki.splunk.com/Community:MoveIndexes (you'll double index until you change the UFs over to point directly at the new server, but this way you'll have more than just the past day of data.) I'll admit however that I did this between two Linux servers, and I'm not sure if between windows and Linux is possible or not.
I would suggest simply enabling the new index on Linux and pointing the forwarder to it. The new index will not contain any data and so should get populated with all data accessible from the forwarder host, whether it is today's data or data that is a week or more old. So you should notice a slight bump in license usage as your new index gets populated with older data.
I have not moved from Windows to Linux with Splunk Enterprise but I am guessing you will not be able to simply move indexes between the two.
Doesn't the forwarder keep track of what it has sent over, though? Or is that all maintained on the indexer side?
(Since both you and acharlieh mentioned it, I was told by a Splunk tech that it is possible, but complicated, to move from Windows to Linux, or vice versa.)
The Forwarder does keep track of the data it has already sent. To have the forwarder resend data for a specific file(s) you will need to use the btprobe command to reset the file(s).
On the Forwarder:
./splunk stop ./btprobe -d ~/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /path/to/file.log --reset ./splunk start
Read more on btprobe here