Recently we configured outputs.conf/props.conf/transforms.conf on our Heavy Forwarders to forward 3 specific events to a remote Syslog collector. The configuration worked for several days, and then we noticed that our daily indexing rate started dropping rapidly from ~ 700GB/day to under 200GB/day. Investigation of splunkd.log on the Heavy Forwarders showed that TcpOutputProc was throwing errors and failing on the connection attempt to our indexers. Engaged Splunk support. Was told that there was a known bug that affected ALL versions such that when using such a setup (forwarding to remote host) that if for any reason the connection is not made as listed in outputs.conf, that ALL forwarding will stop, including forwarding to the indexers!!!
So, we choose not to use outputs.conf from our HF's since any minor connection issues with remote Syslog collectors will cause major problems.
Interested in opinions on other ways to forward filtered events in raw format to a remote collector (ArcSight) in short 5 minute intervals...
This is a known “feature”, not a bug 🙂 Basically because Splunk’s indexing pipeline is shared between the tcpout / syslogout / write to disk functions. When one of these cant send or complete its task, e.g., write to disk or send to tcp, it causes a queue to back up and eventually stop.
Looks at the CEF App for Splunk. There is a CEFOUT command that will fit this usecase specifically, and is where Splunk PS is directing customers to look in use cases as you describe it.
Have been using CEF App for Splunk for ~ 5 months now, and 2 "bugs" identified by the Splunk CEF App team. We do use this app for forwarding CEF events, even though it consistently forwards only 80% or less of the filtered events it should (another "bug" according to Splunk). We are hopeful these issues will be resolved soon.
This requirement is to forward RAW un-formatted data. Would appreciate any suggestions. Your responses are appreciated since for us this is difficult to reproduce
Question1: what about using the same outputs/transforms/props configuration @ the indexers instead of the HF's? Would doing so avoid the "feature" causing indexing to stop, or would the filling up of the queues still ultimately cause the issue?
Question2: Since this is only a problem of filling up of queues, once the remote host starts accepting connections, do the queues now empty?