Deployment Architecture

Best way to verify that sourcetypes are being reported by all systems with SplunkUniversalForwarder installed

rainhailrob
Path Finder

We have several Windows servers with the light SplunkUniversalForwarder installed. Recently we discovered a few servers weren't reporting a sourcetype. I want to verify that each of the servers with a light-SplunkUniversalForwarder installed is sending the appropriate data. I have tried to manually check each system, but that is very tedious and time consuming. If a system isn't reporting a sourcetype then I would like to be alerted or report emailed and then I can troubleshoot more in-depth.

Basically I want to know if all systems are reporting any data for the following:
Perfmon:FreeDiskSpace
Perfmon:LocalNetwork
Perfmon:Memory
Perfmon:CPUTime
WinEventLog:System
WinEventLog:Application
WinEventLog:Security

Does anyone have any suggestions?

0 Karma
1 Solution

kristian_kolb
Ultra Champion

Doing it manually is not too much fun, but there is a problem with doing it in an automated fashion;

How do you determine when a host has stopped sending? Is there a problem, or are the no events being generated? For perfmon, the data should come in regular intervals, the eventlogs can be fairly quiet at times for non-busy servers.

A very simplistic way would be to make a search like this;

index=blah earliest=-1h@h latest=@h | stats dc values(sourcetype) by host | search dc < 7

which would just count the number of different sourcetype, and only keep the results where there are less than 7 distinct sourcetypes seen in the previous hour.

hope this helps,

/k

View solution in original post

mloven_splunk
Splunk Employee
Splunk Employee

I'd venture that as long as a forwarder is sending something, it probably hasn't just stopped sending one sourcetype.

I'd use the Deployment Monitor app here. If I recall correctly, there is a dashboard that deals with something like "Forwarders that haven't sent any data in 2 hours", or something to that effect.

0 Karma

mloven_splunk
Splunk Employee
Splunk Employee

Well, no, of course not. But your original question stated: ..." I want to verify that each of the servers with a light-SplunkUniversalForwarder installed is sending the appropriate data."

0 Karma

rainhailrob
Path Finder

Yes, there is a Deployment App and it has a search for "Forwarder Warnings/Missing Forwarders (A missing forwarder has connected at some point in the past, but has not connected in the last 24 hours.). Unfortunately if the system has never had Splunk installed it won't be in the results. Thank you for the reminder as the app has been helpful as well.

0 Karma

kristian_kolb
Ultra Champion

Doing it manually is not too much fun, but there is a problem with doing it in an automated fashion;

How do you determine when a host has stopped sending? Is there a problem, or are the no events being generated? For perfmon, the data should come in regular intervals, the eventlogs can be fairly quiet at times for non-busy servers.

A very simplistic way would be to make a search like this;

index=blah earliest=-1h@h latest=@h | stats dc values(sourcetype) by host | search dc < 7

which would just count the number of different sourcetype, and only keep the results where there are less than 7 distinct sourcetypes seen in the previous hour.

hope this helps,

/k

View solution in original post

rainhailrob
Path Finder

For Windows servers, I send a test event to app, sec, sys logs: eventcreate /s servername /t Warning /ID 1000 /L APPLICATION /D "Testing splunk"

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!