Getting Data In

How to monitor and alert when the Splunk universal forwarder service has been stopped or modified?

johann2017
Explorer

On my Universal Forwarders, I want to have the ability to monitor and alert off when the Splunk Universal forwarder service has been stopped or modified.
Any options on how to do this?
I am already looking into basic windows event monitoring on windows services, but I didn't know if there was a Splunk related way to do this?
Possibly some Splunk app or something?

0 Karma

Prakash493
Communicator

to monitor the universal forwarder use this query :

index=_internal source=*metrics.log group=tcpin_connections | eval Host=coalesce(hostname, sourcehost) | eval age = (now() - _time ) | stats min(age) as age, max(_time) as LastTime by Host | convert ctime(LastTime) as "Last Active On" | eval Status= case(age < 3600,"Running",age > 3600,"DOWN")|search Status=DOWN

This will show if any universal forwarder is down and will list out the host name and when it was connected last. (Will show if any forwarder is down for more then 1 hr)

0 Karma

johann2017
Explorer

Hello Prakash. I am thinking of this from a security perspective - if a malicious actor is on my network and started turning off my UFs how could I search and alert in a quick amount of time?

0 Karma

Prakash493
Communicator

yes you can modify this part and set how often you need to check the status of forwarder :case(age < 3600,"Running",age > 3600,"DOWN") 3600 sec = 1hr , if you want to see the status in last 10 mins you can change it to 600. And using the query you can set an alert might little tune needed

0 Karma

johann2017
Explorer

How about if a user shuts down their machine and goes home for the evening. Any ideas on how to differentiate a case like that versus someone manually shutting down the UF service?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!