Deployment Architecture

Splunk forwarder loop

dbond
New Member

Hi,

We have a couple of instances where the splunk forwarder gets into a loop due to firewall logging.

The Forwarder is installed on windows 2008 R2, its a domain controller, firewall activity is logged (to the security event log). When The Splunk forwarder sends data to the splunk server it gets logged in the event log, this then triggers another send by splunk, which then get logged and triggered etc. This doesnt always happen, it happens after a reboot, or just after some time, it can be fine.

Why is it doing this? How can it be stopped? I have to stop the forwarder and test after a while to see if it still does it. At the moment it has sent 13GB of logs to splunk, containing mostly logs of the splunk forwarder sending logs to splunk.

Is there a way to get the splunk forwarder to exclude the log for the splunk forwarder, or to only send the data from the logs every 10 seconds, instead of right now when ever a new entry appears? Or is there another solution?

Thank You

David

Tags (1)
0 Karma

Drainy
Champion

Aha, I had the same issue. It is a log message to say that a connection has been established. I think I just needed to turn down the logging for those sorts of messages.
Anyway to filter them and never index them read here;
http://splunk-base.splunk.com/answers/24000/how-do-i-exclude-some-windows-events-from-being-indexed

It has a good example on how to filter by eventcode

Drainy
Champion

Well this is true, but you can install a Splunk indexer on the remote machine and instead configure it as a forwarder, this was how things were done before the UF or where you have specific requirements (such as this), where the power of the Splunk indexer is required.

0 Karma

dbond
New Member

Thanks for your reply. From my understanding, the props.conf and transfoms.conf are only parsed on splunk, the forwarder ingnores them.
This would need to be done on the forwarder as when this happens, thousands of entries a second are added, pushing the CPU usage of splunk forwarder to close to 100%, it appears that the splunk forwarder isnt keeping the connection to the splunk server open.

0 Karma
Get Updates on the Splunk Community!

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...

Index This | What goes away as soon as you talk about it?

May 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...