Getting Data In

What happens to my events at Splunk Light Forwarder when the Indexer goes down?

the_wolverine
Champion

I have a bunch of Lightweight Forwarders (LWF) forwarding to my central indexer. What happens to my events when there's a problem with the indexer and it can't index when my LWFs are trying to send to it?

2 Solutions

dskillman
Splunk Employee
Splunk Employee

The LWF will queue up events and try to resend. There is a maxQueue setting in outputs.conf that you can configure a larger queue. There are other settings you can tweak to cover your scenarios like dropping events or blocking if the queue fills. I would recommend spinning up another Splunk Indexer and use AutoLB and distributed search to limit losing connectivity to the indexing tier. You'll get better redundancy and better performance.

http://www.splunk.com/base/Documentation/4.1/Admin/Outputsconf

View solution in original post

Dan
Splunk Employee
Splunk Employee

If the output queue fills up, all preceding Splunk processors will block. This means if you're monitoring a file or directory, the tailing processor will block and stop moving the pointers into each file. Once the indexer is up and the output queue empties, the tailing processor will unblock and the pointer will eventually catch up. You shouldn't lose data, unless the outage is so long that the file gets rolled or deleted.

If you have network inputs, no such luck.

View solution in original post

Dan
Splunk Employee
Splunk Employee

If the output queue fills up, all preceding Splunk processors will block. This means if you're monitoring a file or directory, the tailing processor will block and stop moving the pointers into each file. Once the indexer is up and the output queue empties, the tailing processor will unblock and the pointer will eventually catch up. You shouldn't lose data, unless the outage is so long that the file gets rolled or deleted.

If you have network inputs, no such luck.

Dan
Splunk Employee
Splunk Employee

that being said, I also recommend multiple Splunk Indexers and AutoLB. You get auto-failover in bad situations and awesome performance all other times.

0 Karma

Dan
Splunk Employee
Splunk Employee

like all else, this is configurable. dropEventsOnQueueFull in outputs.conf

0 Karma

dskillman
Splunk Employee
Splunk Employee

The LWF will queue up events and try to resend. There is a maxQueue setting in outputs.conf that you can configure a larger queue. There are other settings you can tweak to cover your scenarios like dropping events or blocking if the queue fills. I would recommend spinning up another Splunk Indexer and use AutoLB and distributed search to limit losing connectivity to the indexing tier. You'll get better redundancy and better performance.

http://www.splunk.com/base/Documentation/4.1/Admin/Outputsconf

Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...