Splunk Enterprise

Events where a value is missing

jeremyhagand61
Communicator

Hi,

I'm trying to use Splunk to provide a report on servers where a service is absent. So I have one event per service per host. So if there are 10 services running on 1 host, that is 10 different events. My idea was to do a search which combines all of the services on a host into a single field and then search where that field doesn't contain the value I am looking for, but I have no idea how to achieve this.

Here are a couple of sample raw events from the same host

 

20200702162757.583428
Caption=Remote Desktop Configuration
Description=Remote Desktop Configuration service (RDCS) is responsible for all Remote Desktop Services and Remote Desktop related configuration and session maintenance activities that require SYSTEM context. These include per-session temporary folders, RD themes, and RD certificates.
Name=SessionEnv
PathName=C:\WINDOWS\System32\svchost.exe -k netsvcs
StartMode=Manual
StartName=localSystem
State=Running
Status=OK
wmi_type=Service

20200702162757.583428
Caption=Symantec Endpoint Protection WSC Service
Description=Allows Symantec Endpoint Protection to report status to the Windows Security Center.
Name=sepWscSvc
PathName="C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\14.3.558.0000.105\Bin64\sepWscSvc64.exe"
StartMode=Auto
StartName=LocalSystem
State=Running
Status=OK
wmi_type=Service

 

 Assume I want to return hosts where the second service entry is absent.

Labels (1)
Tags (1)
0 Karma
1 Solution

jeremyhagand61
Communicator

I have tweaked this slightly to make it better.

source="WMI:Service"
| transaction maxspan=1m host
| dedup host,Caption
| regex Caption!="Symantec Endpoint Protection"
| table host,Caption

View solution in original post

0 Karma

jeremyhagand61
Communicator

I have tweaked this slightly to make it better.

source="WMI:Service"
| transaction maxspan=1m host
| dedup host,Caption
| regex Caption!="Symantec Endpoint Protection"
| table host,Caption

View solution in original post

0 Karma

bowesmana
SplunkTrust
SplunkTrust

You are using the transaction command, which can have memory issues for long running transactions. From your solution, it looks like you're expecting Symantec Endpoint Protection to touch base every minute. An alternative approach avoiding the transaction command could be

source="WMI:Service"
| bin _time span=1m
| stats values(Caption) as Captions by _time host
| where isnull(mvfind(Captions,"Symantec Endpoint Protection"))

which is asking to find all Captions for a host per minute, where there is no Symantec Endpoint Protection

it then will show you the Captions for that host within that minute for all minutes where there was no reqiured Caption. 

0 Karma

jeremyhagand61
Communicator

I actually managed to solve this as follows:

source="WMI:Service"
| dedup host Caption
| transaction maxspan=60m host
| table host,Caption
| regex Caption!="Symantec Endpoint Protection"
Tags (1)
0 Karma

renjith_nair
SplunkTrust
SplunkTrust

@jeremyhagand61,

We need to have a master set to compare against the events to find the "missing" service. Lookup is one of the most used method used in these cases.

  • Create a lookup file with the server and service combination

e.g.

 

server,service
host1,service1
host1,service2
host1,service3
host2,service1
host2,service2
host2,service3
host3,service1
host3,service2
host3,service3

 

  • Compare against that with the events
  • Find the missing entries

 

index="your index" "other search parameters"
|stats count by server,service
|inputlookup servers.csv append=true| fillnull count
|stats sum(count) as count by server,service

 

For those who have 0 count is missing one or more services

where count < 1
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!