Getting Data In

Where is the data stored on a Forwarder when the Receiver is down

Engager

I know the forwarder will buffer its data if the receiver goes down for some reason.Where is the data stored(directory and/or folder)?

Is it different for windows and linux forwarders.

Thanks,

Tags (1)

Splunk Employee
Splunk Employee

Edit: Was your question more along the lines of where the buffered forwarder data is stored? In 3.x days we used to have a persistent queue, but that is deprecated and we do not recommend using it in 4.x See this
http://answers.splunk.com/questions/1437/i-want-to-configure-usepersistentqueue-so-splunk-will-buffe...

Otherwise, forwarder data wont get stored locally if your indexer goes down, it will buffer a certain # of events based on your outputs.conf settings.
maxQueueSize =

Having at least one indexer up would be a good reason to go with a AutoLB or Data clone Environment setup. In both these environments the data will go to your other indexers. http://www.splunk.com/base/Documentation/4.0.10/Installation/CapacityplanningforalargerSplunkdeploym...

Splunk Employee
Splunk Employee

This is also why you don't want to have a persistentQueue on any forwarder that's tailing files. The queue will just read in the whole file, which is pointless, since the file itself is its own queue.

It can make sense to use a larger and/or persistent queue if your forwarder is collecting only remote information that it can't go get later (e.g., syslog UDP, polling for instantaneous statistics).

0 Karma

Splunk Employee
Splunk Employee

Yes, if the forwarder goes down, items in the forwarder queue will be lost. However, if you're tailing local files, I believe the pointers won't be updated till the data is sent. So no. For a typical light forwarder, this isn't a problem. If you're collecting network data however, yes, it will be lost since the forwarder can't go back and get it again.

0 Karma

Super Champion

So if the forwarder goes down after the indexer goes down, then events are dropped? Is that what I'm hearing?

0 Karma

Splunk Employee
Splunk Employee

Yes indeed, MaxQueueSize settings buffer events in memory The default maxQueueSize requires approximately 100 MB of memory If you wish to increase your MaxQueueSize to handle 1,000,000 events it will require 1 GB of memory

0 Karma

Splunk Employee
Splunk Employee

Buffered in memory?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!