Deployment Architecture

Forwarder won't stop forwarding

mmoermans
Path Finder

A strange issue is happening, a few of our forwarders are sending a massive amount of data (wineventlog:security) to Splunk. I've tried to remove the Windows_TA_Splunk from the forwarders but they keep sending their eventlogs regardless.

I've tried restarting them several times but they don't stop sending data even though they don't have any received apps. Anyone know how to make them stop sending the wineventlog?

0 Karma
1 Solution

mmoermans
Path Finder

Solved by making a copy of Splunk_TA_Windows called Splunk_TA_Vindows (Using the letter V since it's earlier in alphabet and would take precedent) and disabling wineventlog:security in it, pushing that to the forwarder.

Even though I had removed the client from the Serverclass for windows eventlog it hadn't removed the app for some reason and only stopped after pushing a new app (Splunk_TA_Vindows) above it which disabled the winsecurity logging.

Seems to be a bug for forwarders on 6.3.1., best fix would be to update the forwarder but wasn't possible in this case.

View solution in original post

0 Karma

mmoermans
Path Finder

Solved by making a copy of Splunk_TA_Windows called Splunk_TA_Vindows (Using the letter V since it's earlier in alphabet and would take precedent) and disabling wineventlog:security in it, pushing that to the forwarder.

Even though I had removed the client from the Serverclass for windows eventlog it hadn't removed the app for some reason and only stopped after pushing a new app (Splunk_TA_Vindows) above it which disabled the winsecurity logging.

Seems to be a bug for forwarders on 6.3.1., best fix would be to update the forwarder but wasn't possible in this case.

View solution in original post

0 Karma

horsefez
SplunkTrust
SplunkTrust

Could you go into a bit more detail on how you managed it? Thanks!

0 Karma

ddrillic
Ultra Champion

I had a similar experience recently at Where does the forwarder enqueue files?

My experience was -

-- It is the universal forwarder reading the files. I think it's TailReader versus BatchReader. What I see is that TailReader is real-time versus BatchReader which is not and also we don't seem to have control of the pending batches.

So, when the forwarder was in this BatchReader mode, the only way to stop it was to uninstall and reinstall the forwarder.

Maybe you are in a similar situation...

0 Karma

jgbricker
Contributor

Are they deployments clients? If so they would keep getting updated with the enabled inputs settings even though you are manually changing at the source. You may need to make a new copy of the TA and and setup a separate server class excluding those from the main server class and including in the non security input server class.

somesoni2
Revered Legend

Try to run btool command to see if that eventlog monitoring is enabled elsewhere.

bin/splunk cmd btool inputs list wineventlog:security --debug

If it shows enabled at path other than TA apps, disabled from that place as well (and restart)

.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!