topic Re: How to forward old data from a forwarder to a new Splunk index? in Getting Data In

How to forward old data from a forwarder to a new Splunk index?

LuiesCui — Sat, 28 Feb 2015 03:00:11 GMT

Hey guys, I'm new to splunk and I really need ur help!!!

As what I know, once the data from a .log file are loaded by forwarder to Splunk, the forwarder won't load them again if the .log file has not been change. What I want to do is to get the old data to a new Splunk index without changing the .log file and what happens is the index gets nothing. So is there any way to clear the record (or change some configuration? i don't know) on what the forwarder has already sent and make it forward that old data again? Thx so much!!!

Re: How to forward old data from a forwarder to a new Splunk index?

MuS — Sat, 28 Feb 2015 07:42:26 GMT

Hi LuiesCui,

you can either clean the index fishbucket on the forwarder by running the following command:

$SPLUNK_HOME/bin/splunk clean all

This will force the forwarder the re-load everything or you add the following option crcSalt = REINDEXMEPLEASE to your monitor stanza in inputs.conf:

[monitor://C:\temp\tutorialdata]
disabled = 0
index = foo
sourcetype = myfoo
crcSalt = REINDEXMEPLEASE

restart the forwarder, wait some seconds and immediately remove the option from inputs.conf without restarting the forwarder.
This will enable you to selectively re-load single inputs. Both methods tested and working on Windows with Splunk universal forwarder 6.1.1

Hope that helps ...

cheers, MuS

Re: How to forward old data from a forwarder to a new Splunk index?

LuiesCui — Mon, 02 Mar 2015 01:48:32 GMT

Thank u so much! Still I have some questions about the fishbucket. I'm using the forwarder on win 7 and I think where I type the command is the cmd.exe right? But what cmd.exe shows is "is not recognized as an internal or external command,
operable program or batch file." And the way by editing input.conf works once. If I do the same thing on the same .log files again, no data loaded.

Re: How to forward old data from a forwarder to a new Splunk index?

MuS — Mon, 02 Mar 2015 06:45:14 GMT

On windows it's probably c:\program files\spunkforwarder\bin\splunk clean eventdata -index _thefishbucket . The inputs.conf method will work each time, don't forget to restart the forwarder after the modification.

Re: How to forward old data from a forwarder to a new Splunk index?

bohrasaurabh — Mon, 02 Mar 2015 18:43:45 GMT

Another approach would be backup the c:\program files\spunkforwarder\etc\apps directory on the forwarder machine and then un-install and reinstall the forwarder. place back the contents of the apps directory and then start the forwarder.

Re: How to forward old data from a forwarder to a new Splunk index?

LuiesCui — Tue, 03 Mar 2015 01:54:25 GMT

Well the input.conf method is still not working. And the cli method shows "Error: Cleaning eventdata is not supported on this version." I am using the forwarder of 6.1.1 32bit on win 7 64bit. What would happen if I just delete the fishbucket?

Re: How to forward old data from a forwarder to a new Splunk index?

LuiesCui — Tue, 03 Mar 2015 01:55:21 GMT

Can I just delete the fishbucket?

Re: How to forward old data from a forwarder to a new Splunk index?

MuS — Tue, 03 Mar 2015 10:05:19 GMT

my bad, sorry the clean command was from an indexer ..... see my updates in the answer, both will work for you.

Re: How to forward old data from a forwarder to a new Splunk index?

MuS — Tue, 03 Mar 2015 10:28:40 GMT

This is some kind of a hardcore way, but it will work as well if you stop the forwarder and delete C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket the directory will then be re-created if you start the forwarder again.

Re: How to forward old data from a forwarder to a new Splunk index?

LuiesCui — Wed, 04 Mar 2015 01:19:13 GMT

Thank you so much!

Re: How to forward old data from a forwarder to a new Splunk index?

dantimola — Wed, 14 Jun 2017 05:01:23 GMT

Hi, MuS,

What will happen if he didn't remove the option from inputs.conf?

Thanks,
Dan

Re: How to forward old data from a forwarder to a new Splunk index?

MuS — Wed, 14 Jun 2017 19:55:33 GMT

Hi dantimola,

if you don't remove it, the universal forwarder will re-index all the things all the time for this monitor. By adding it once and removing it immediately the uf will just re-index all the things once.

Hope this answers your question?

cheers, MuS