Getting Data In

Where is the data stored on a Forwarder when the Receiver is down

skibum
Engager

I know the forwarder will buffer its data if the receiver goes down for some reason.Where is the data stored(directory and/or folder)?

Is it different for windows and linux forwarders.

Thanks,

Tags (1)

Chris_R_
Splunk Employee
Splunk Employee

Edit: Was your question more along the lines of where the buffered forwarder data is stored? In 3.x days we used to have a persistent queue, but that is deprecated and we do not recommend using it in 4.x See this
http://answers.splunk.com/questions/1437/i-want-to-configure-usepersistentqueue-so-splunk-will-buffe...

Otherwise, forwarder data wont get stored locally if your indexer goes down, it will buffer a certain # of events based on your outputs.conf settings.
maxQueueSize =

Having at least one indexer up would be a good reason to go with a AutoLB or Data clone Environment setup. In both these environments the data will go to your other indexers. http://www.splunk.com/base/Documentation/4.0.10/Installation/CapacityplanningforalargerSplunkdeploym...

gkanapathy
Splunk Employee
Splunk Employee

This is also why you don't want to have a persistentQueue on any forwarder that's tailing files. The queue will just read in the whole file, which is pointless, since the file itself is its own queue.

It can make sense to use a larger and/or persistent queue if your forwarder is collecting only remote information that it can't go get later (e.g., syslog UDP, polling for instantaneous statistics).

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Yes, if the forwarder goes down, items in the forwarder queue will be lost. However, if you're tailing local files, I believe the pointers won't be updated till the data is sent. So no. For a typical light forwarder, this isn't a problem. If you're collecting network data however, yes, it will be lost since the forwarder can't go back and get it again.

0 Karma

Lowell
Super Champion

So if the forwarder goes down after the indexer goes down, then events are dropped? Is that what I'm hearing?

0 Karma

Chris_R_
Splunk Employee
Splunk Employee

Yes indeed, MaxQueueSize settings buffer events in memory The default maxQueueSize requires approximately 100 MB of memory If you wish to increase your MaxQueueSize to handle 1,000,000 events it will require 1 GB of memory

0 Karma

Dan
Splunk Employee
Splunk Employee

Buffered in memory?

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...