How to Produce a Log for "Splunk Forwarder Removal"

[zekiramhi]

Hello Splunkers,

I would like to keep track of my machines that contain a Splunk forwarder by producing a log of their removal, is it actually possible by any means?

Also, is there a way to set a password for the forwarder to disable any uninstallation attempts?

All help is appreciated, Thanks!

[nickhills · Accepted solution]

The problem is ... timing...

Splunk is not the correct tool for monitoring in real time for the removal of 'Splunk Forwarders' - why?
Because the second you stop/break/uninstall Splunk you stop seeing events from that host, and in 99% of cases, the log which records Splunk was removed, will only be written AFTER the Splunk process has stopped.

Because of this, you need to tackle the issue differently.
Your first aim should be prevent uninstallation of the tool - and making sure users are running is least privilege mode (ie, not as admins) wins you most of that war.

A second approach is to automatically reinstall missing applications when they are removed, but if your user has Admin rights, this becomes a game of 'cat and mouse'

The world however, is not perfect, and sometimes local admin rights may be necessary evil for many people (although there is ALWAYS another way) so if you can't prevent admins getting up to mischief, your next best bet is retrospectively detecting when they have been.

So the third approach is to look for machines which have previously sent events, but have now stopped. There are some pitfalls with this approach, such as laptops which are not on all the time, so you have to look at the numbers subjectively - unless you have another source which can tell you categorically that a machine is really on the network (DHCP/Forescout/Firewall Logs/CMDB discovery tools etc)

If your forwarders are managed by a Deployment Server, the DS can show you clients which haven't connected for a while.
The DMC call also show you missing forwarders
Finally this app is very handy for finding other forwarder issues https://splunkbase.splunk.com/app/3805/

In short, this sounds like a people problem - and not an fun one if you can't trust your privileged users.

[lakshman239]

One way would do add the splunk install folder to 'files and folder audit' and when its removed, it will create an entry, which can be monitored using any monitoring tools in your organization

[zekiramhi]

So basically you are advising to place the install folder to the Windows file auditing logs, can't we keep track of that in Splunk rather than any monitoring device? or what do you exactly mean by monitoring device.

Would appreciate abit more clarification please.