Indexer down: On "Pause the Monitoring", will forwarders pick up where they left off once the indexer is online?

[khhenderson]

We have a small Splunk infrastructure, one indexer, one search head and 300 machines with forwarders installed. Our indexer has gone down with hardware issues. Our log traffic is less than 10GB a day over the weekends and up to 60GB during the weekdays. Our indexer went down on Saturday night. From what I've read, "if the indexer refuses data (full or down) then the forwarders fill their memory queue up to 2MB (default), then pause the monitoring".

My question is on "Pause the Monitoring", will the forwarders pick up where they left off once the indexer is online?

[woodcock]

Yes, if the forwarder is a monitor-type input. Even then, the files may rotate out from under Splunk or housekeeping processes may delete them. Also, if it is another type, like port monitor (which you really should not be doing), then it will have no choice but to drop events once the buffer fills.

[sudosplunk · Accepted solution]

Hi @khhenderson,

Yes. UF will pick up from where it left off with the help of _thefishbucket index. It contains seek pointers and CRCs for the files you are indexing, so splunkd can tell if it has read them already.

You can find more info about _thefishbucket and it's operation here. Hope this helps!

[khhenderson]

@nittala_surya the _thefishbucket index is on the indexer? I can check once we get the machine to boot up.

[sudosplunk]

No. It is on the universal forwarder. The location of fishbucket on forwarder is, $SPLUNK_HOME/var/lib/splunk/fishbucket.

And I just realized that, Splunk no longer lets you look at the fishbucket index but it's purpose is still the same (to keep track of pointers).

From another accepted answer,

The big problem with the fishbucket stuff that Andrea wrote about is that it does not apply in 4.x and up. It's accurate if you have a 3.x forwarder, but 4.x no longer stores the data in a Splunk index (it wasn't a good idea in the first place, though it was convenient for some purposes), but rather in the splunk_private_db inside the fishbucket index location. You can kind of examine the data using the $SPLUNK_HOME/bin/btprobe tool, but it's not that helpful, in particular because we are now only storing the hash and position, and not recording any of the other information that used to be in the fishbucket index.

I think there might be some plans to add back some tools and info to get some of this functionality back, but you might want to file ERs on it.

I updated my answer above.