topic Re: How to reindex data from a forwarder in Getting Data In

How to reindex data from a forwarder

mataharry — Fri, 18 Jan 2013 23:39:28 GMT

I have a Storm project and I want to clean all and reindex only the last days, and some specific files.
I have Splunk Universal forwarders monitoring my files for now.

I suppose that this is similar for a Splunk Enterprise, when we clear an index and Storm when we manually empty a project.

Re: How to reindex data from a forwarder

yannK — Mon, 28 Sep 2020 13:08:36 GMT

First of all even before reindexing, to configure Splunk to index only recent data, you can use the 2 techniques:

for file monitoring, add the parameter ignoreOlderThan in inputs.conf
It will look at the modtime of the files, example : ignoreOlderThan=7d will index only files touched during the last 7 days. On linux you can couple this with the touch command to change the modtime of a file and trigger the indexing.
see http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf
for WinEventLogs, you can setup the parameter current_only=1 in inputs.conf to exclude the historical logs, and starts only now.

Now that you have setup your inputs to avoid blastering your instance, you can focus on How to force a splunk instance to reindex a file that has already be indexed.

the radical method is to clean the fishbucket index . That will remove the memory of every files, But it will reindex all.
on an indexer splunk clean eventdata -index _fishbucket
on a forwarder by removing the folder $SPLUNK_HOME/var/lib/splunk/fishbucket
or selectivelly forgot a single file from the fishbucket

splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file $FILE --reset
manually reindex each file with the oneshot option,
you also can edit the log file and add a comment on the first line that will force the file to detected as a new file.

./splunk add oneshot "/path/to/my/file.log" -sourcetype mysourcetype
modify the first line of the files to reindex, by default splunk checks the first 256 chars of a file to differentiate them. If you had a simple comment on the first line it wil reindex it
change the crcSalt, create a new input for a new folder, add all the correct sourcetypes, etc...
using a static string that will force a one time reindexing.

crcSalt= REINDEXMEPLEASE

or add the option

crcSalt= <SOURCE>

then move or copy the files to be reindex to the folder, they will be detected as new (because the path will be considered in the crc calculation). (ps the source field will be different of course.)

see http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf

Remark : before reindexing you may want to remove the existing data in splunk to avoid duplicates.

You can empty the index if nothing has to be preserved.
Or use the |delete command to selectively hide some events. see http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Delete

Remark : if you are monitoring windows logs (wineventlog) or are using modular inputs, the counters are not in the fishbucket.
you need to clear the checkpoints files in $SPLUNK_HOME/var/lib/splunk/modinputs/

Re: How to reindex data from a forwarder

coleman07 — Tue, 30 Jun 2015 20:57:26 GMT

This method doesn't work with the Splunk 6 Forwarder but I found if you remove all directories in C:\Program Files\SplunkUniversalForwarder\var\lib\splunk, this will force Splunk to reindex all the Window's logs. You have to remove all of them.

Re: How to reindex data from a forwarder

bnorthway — Tue, 07 Jul 2015 12:44:06 GMT

FYI the Splunk service must be stopped before removing files from the fishbucket

Re: How to reindex data from a forwarder

season88481 — Thu, 21 Apr 2016 01:41:20 GMT

Hi I am using Splunk 6.4, just use:
splunk clean eventdata -index _fishbucket

'-index' is no longer required.

Re: How to reindex data from a forwarder

MuS — Tue, 23 Aug 2016 02:31:00 GMT

$SPLUNK_HOME/bin/splunk clean all still works on 6.4.2 😉

Re: How to reindex data from a forwarder

slierninja — Fri, 09 Sep 2016 00:27:43 GMT

To re-index forwarded events for Windows Event logs, I had to remove the modinputs file that bookmarks the last RecordId sent to splunk indexer....I also deleted the fishbuckets as suggested by @yannK

Source Path: SplunkUniversalForwarder\var\lib\splunk\modinputs\WinEventLog

<BookmarkList>
  <Bookmark Channel='application' RecordId='48426' IsCurrent='true'/>
</BookmarkList>

Removing the application file and restarting the forwarder will force a reindex for windows event logs.

Re: How to reindex data from a forwarder

sloshburch — Fri, 28 Jul 2017 19:20:47 GMT

FYI. Now it's _thefishbucket, not _fishbucket. At least in 6.6.2...

Re: How to reindex data from a forwarder

jgreen12 — Tue, 21 Nov 2017 19:57:54 GMT

Will any of these methods work for re-indexing the data from an API? Many of the resources I've found only mention log files when speaking of re-indexing. My data input is an API. I am able to clean the index for this API, but want to ensure I can re-index all the data.

Re: How to reindex data from a forwarder

gjanders — Tue, 21 Nov 2017 22:16:27 GMT

jgreen12 please open a new question as this question is answered (and very old)

It may or may not relate to modular inputs and there may be a checkpoint file keeping track of the data it has obtained but a new question would make more sense here rather than guessing...

Re: How to reindex data from a forwarder

sloshburch — Wed, 22 Nov 2017 14:08:25 GMT

Agreed. Also, you may need to check with the creator of that particular add-on. Once you create the new question thread, link to it here and we can jump over there.

Re: How to reindex data from a forwarder

codebuilder — Fri, 07 Feb 2020 16:18:44 GMT

You can also trigger the forwarder to re-ingest files by adding/modifying the crcsalt parameter within inputs.conf.

crcSalt = <string>
* Use this setting to force the input to consume files that have matching CRCs
  (cyclic redundancy checks).
    * By default, the input only performs CRC checks against the first 256
      bytes of a file. This behavior prevents the input from indexing the same
      file twice, even though you might have renamed it, as with rolling log
      files, for example. Because the CRC is based on only the first
      few lines of the file, it is possible for legitimately different files
      to have matching CRCs, particularly if they have identical headers.
* If set, <string> is added to the CRC.
* If set to the literal string "<SOURCE>" (including the angle brackets), the
  full directory path to the source file is added to the CRC. This ensures
  that each file being monitored has a unique CRC. When 'crcSalt' is invoked,
  it is usually set to <SOURCE>.
* Be cautious about using this setting with rolling log files; it could lead
  to the log file being re-indexed after it has rolled.
* In many situations, 'initCrcLength' can be used to achieve the same goals.
* Default: empty string.

Re: How to reindex data from a forwarder

lyeager — Wed, 07 Dec 2022 18:30:35 GMT

In 8.2.6, the btprobe command is missing:

$ ./bin/splunk help commands | grep cmd
    cmd [btool|exporttool|importtool|locktest|locktool|parsetest|pcregextest|signtool|walklex]

EDIT: nevermind, yes it is. It's just not listed in the help menu.

Re: How to reindex data from a forwarder

gabmsmith — Tue, 23 Apr 2024 20:17:36 GMT

Would it work to replace the Universal Forwarder with a new one?

Re: How to reindex data from a forwarder

gabmsmith — Wed, 24 Apr 2024 14:55:07 GMT

This didn't work...

Re: How to reindex data from a forwarder

R15 — Tue, 01 Oct 2024 20:54:53 GMT

This list has aged quite a bit, is it still accurate? @yannK

Re: How to reindex data from a forwarder

yannK — Tue, 01 Oct 2024 21:18:54 GMT

@R15 For monitoring Stanzas, it's still pretty much the same.
However, many new type of inputs exists too (modular, scripted, HEC etc...), who do not rely on the fishbucket.