topic Re: Forwarder won't stop forwarding in Deployment Architecture

Forwarder won't stop forwarding

mmoermans — Tue, 29 Sep 2020 16:36:13 GMT

A strange issue is happening, a few of our forwarders are sending a massive amount of data (wineventlog:security) to Splunk. I've tried to remove the Windows_TA_Splunk from the forwarders but they keep sending their eventlogs regardless.

I've tried restarting them several times but they don't stop sending data even though they don't have any received apps. Anyone know how to make them stop sending the wineventlog?

Re: Forwarder won't stop forwarding

somesoni2 — Thu, 02 Nov 2017 15:31:36 GMT

Try to run btool command to see if that eventlog monitoring is enabled elsewhere.

bin/splunk cmd btool inputs list wineventlog:security --debug

If it shows enabled at path other than TA apps, disabled from that place as well (and restart)

Re: Forwarder won't stop forwarding

jgbricker — Fri, 03 Nov 2017 00:28:06 GMT

Are they deployments clients? If so they would keep getting updated with the enabled inputs settings even though you are manually changing at the source. You may need to make a new copy of the TA and and setup a separate server class excluding those from the main server class and including in the non security input server class.

Re: Forwarder won't stop forwarding

ddrillic — Fri, 03 Nov 2017 01:01:40 GMT

I had a similar experience recently at Where does the forwarder enqueue files?

My experience was -

-- It is the universal forwarder reading the files. I think it's TailReader versus BatchReader. What I see is that TailReader is real-time versus BatchReader which is not and also we don't seem to have control of the pending batches.

So, when the forwarder was in this BatchReader mode, the only way to stop it was to uninstall and reinstall the forwarder.

Maybe you are in a similar situation...

Re: Forwarder won't stop forwarding

mmoermans — Tue, 29 Sep 2020 16:33:40 GMT

Solved by making a copy of Splunk_TA_Windows called Splunk_TA_Vindows (Using the letter V since it's earlier in alphabet and would take precedent) and disabling wineventlog:security in it, pushing that to the forwarder.

Even though I had removed the client from the Serverclass for windows eventlog it hadn't removed the app for some reason and only stopped after pushing a new app (Splunk_TA_Vindows) above it which disabled the winsecurity logging.

Seems to be a bug for forwarders on 6.3.1., best fix would be to update the forwarder but wasn't possible in this case.

Re: Forwarder won't stop forwarding

horsefez — Fri, 03 Nov 2017 08:52:48 GMT

Could you go into a bit more detail on how you managed it? Thanks!