Forwarder won't stop forwarding

[mmoermans]

A strange issue is happening, a few of our forwarders are sending a massive amount of data (wineventlog:security) to Splunk. I've tried to remove the Windows_TA_Splunk from the forwarders but they keep sending their eventlogs regardless.

I've tried restarting them several times but they don't stop sending data even though they don't have any received apps. Anyone know how to make them stop sending the wineventlog?

[mmoermans · Accepted solution]

Solved by making a copy of Splunk_TA_Windows called Splunk_TA_Vindows (Using the letter V since it's earlier in alphabet and would take precedent) and disabling wineventlog:security in it, pushing that to the forwarder.

Even though I had removed the client from the Serverclass for windows eventlog it hadn't removed the app for some reason and only stopped after pushing a new app (Splunk_TA_Vindows) above it which disabled the winsecurity logging.

Seems to be a bug for forwarders on 6.3.1., best fix would be to update the forwarder but wasn't possible in this case.

[horsefez]

Could you go into a bit more detail on how you managed it? Thanks!

[ddrillic]

I had a similar experience recently at Where does the forwarder enqueue files?

My experience was -

-- It is the universal forwarder reading the files. I think it's TailReader versus BatchReader. What I see is that TailReader is real-time versus BatchReader which is not and also we don't seem to have control of the pending batches.

So, when the forwarder was in this BatchReader mode, the only way to stop it was to uninstall and reinstall the forwarder.

Maybe you are in a similar situation...

[jgbricker]

Are they deployments clients? If so they would keep getting updated with the enabled inputs settings even though you are manually changing at the source. You may need to make a new copy of the TA and and setup a separate server class excluding those from the main server class and including in the non security input server class.

[somesoni2]

Try to run btool command to see if that eventlog monitoring is enabled elsewhere.

bin/splunk cmd btool inputs list wineventlog:security --debug

If it shows enabled at path other than TA apps, disabled from that place as well (and restart)