Deployment Architecture

Splunk forwarder loop

dbond
New Member

Hi,

We have a couple of instances where the splunk forwarder gets into a loop due to firewall logging.

The Forwarder is installed on windows 2008 R2, its a domain controller, firewall activity is logged (to the security event log). When The Splunk forwarder sends data to the splunk server it gets logged in the event log, this then triggers another send by splunk, which then get logged and triggered etc. This doesnt always happen, it happens after a reboot, or just after some time, it can be fine.

Why is it doing this? How can it be stopped? I have to stop the forwarder and test after a while to see if it still does it. At the moment it has sent 13GB of logs to splunk, containing mostly logs of the splunk forwarder sending logs to splunk.

Is there a way to get the splunk forwarder to exclude the log for the splunk forwarder, or to only send the data from the logs every 10 seconds, instead of right now when ever a new entry appears? Or is there another solution?

Thank You

David

Tags (1)
0 Karma

Drainy
Champion

Aha, I had the same issue. It is a log message to say that a connection has been established. I think I just needed to turn down the logging for those sorts of messages.
Anyway to filter them and never index them read here;
http://splunk-base.splunk.com/answers/24000/how-do-i-exclude-some-windows-events-from-being-indexed

It has a good example on how to filter by eventcode

Drainy
Champion

Well this is true, but you can install a Splunk indexer on the remote machine and instead configure it as a forwarder, this was how things were done before the UF or where you have specific requirements (such as this), where the power of the Splunk indexer is required.

0 Karma

dbond
New Member

Thanks for your reply. From my understanding, the props.conf and transfoms.conf are only parsed on splunk, the forwarder ingnores them.
This would need to be done on the forwarder as when this happens, thousands of entries a second are added, pushing the CPU usage of splunk forwarder to close to 100%, it appears that the splunk forwarder isnt keeping the connection to the splunk server open.

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...