Deployment Architecture

Forward raw data to remote host - avoiding outputs.conf bug

tlmayes
Contributor

Recently we configured outputs.conf/props.conf/transforms.conf on our Heavy Forwarders to forward 3 specific events to a remote Syslog collector. The configuration worked for several days, and then we noticed that our daily indexing rate started dropping rapidly from ~ 700GB/day to under 200GB/day. Investigation of splunkd.log on the Heavy Forwarders showed that TcpOutputProc was throwing errors and failing on the connection attempt to our indexers. Engaged Splunk support. Was told that there was a known bug that affected ALL versions such that when using such a setup (forwarding to remote host) that if for any reason the connection is not made as listed in outputs.conf, that ALL forwarding will stop, including forwarding to the indexers!!!

So, we choose not to use outputs.conf from our HF's since any minor connection issues with remote Syslog collectors will cause major problems.

Interested in opinions on other ways to forward filtered events in raw format to a remote collector (ArcSight) in short 5 minute intervals...

Tags (1)
0 Karma

esix_splunk
Splunk Employee
Splunk Employee

This is a known “feature”, not a bug 🙂 Basically because Splunk’s indexing pipeline is shared between the tcpout / syslogout / write to disk functions. When one of these cant send or complete its task, e.g., write to disk or send to tcp, it causes a queue to back up and eventually stop.

Looks at the CEF App for Splunk. There is a CEFOUT command that will fit this usecase specifically, and is where Splunk PS is directing customers to look in use cases as you describe it.

See here : https://splunkbase.splunk.com/app/1847/

tlmayes
Contributor

Have been using CEF App for Splunk for ~ 5 months now, and 2 "bugs" identified by the Splunk CEF App team. We do use this app for forwarding CEF events, even though it consistently forwards only 80% or less of the filtered events it should (another "bug" according to Splunk). We are hopeful these issues will be resolved soon.

This requirement is to forward RAW un-formatted data. Would appreciate any suggestions. Your responses are appreciated since for us this is difficult to reproduce

Question1: what about using the same outputs/transforms/props configuration @ the indexers instead of the HF's? Would doing so avoid the "feature" causing indexing to stop, or would the filling up of the queues still ultimately cause the issue?

Question2: Since this is only a problem of filling up of queues, once the remote host starts accepting connections, do the queues now empty?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...