topic Re: Events where a value is missing in Splunk Enterprise

Events where a value is missing

jeremyhagand61 — Thu, 02 Jul 2020 06:40:03 GMT

Hi,

I'm trying to use Splunk to provide a report on servers where a service is absent. So I have one event per service per host. So if there are 10 services running on 1 host, that is 10 different events. My idea was to do a search which combines all of the services on a host into a single field and then search where that field doesn't contain the value I am looking for, but I have no idea how to achieve this.

Here are a couple of sample raw events from the same host

20200702162757.583428 Caption=Remote Desktop Configuration Description=Remote Desktop Configuration service (RDCS) is responsible for all Remote Desktop Services and Remote Desktop related configuration and session maintenance activities that require SYSTEM context. These include per-session temporary folders, RD themes, and RD certificates. Name=SessionEnv PathName=C:\WINDOWS\System32\svchost.exe -k netsvcs StartMode=Manual StartName=localSystem State=Running Status=OK wmi_type=Service 20200702162757.583428 Caption=Symantec Endpoint Protection WSC Service Description=Allows Symantec Endpoint Protection to report status to the Windows Security Center. Name=sepWscSvc PathName="C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\14.3.558.0000.105\Bin64\sepWscSvc64.exe" StartMode=Auto StartName=LocalSystem State=Running Status=OK wmi_type=Service

Assume I want to return hosts where the second service entry is absent.

Re: Events where a value is missing

renjith_nair — Thu, 02 Jul 2020 07:46:55 GMT

@jeremyhagand61,

We need to have a master set to compare against the events to find the "missing" service. Lookup is one of the most used method used in these cases.

Create a lookup file with the server and service combination

e.g.

server,service host1,service1 host1,service2 host1,service3 host2,service1 host2,service2 host2,service3 host3,service1 host3,service2 host3,service3

Compare against that with the events
Find the missing entries

index="your index" "other search parameters" |stats count by server,service |inputlookup servers.csv append=true| fillnull count |stats sum(count) as count by server,service

For those who have 0 count is missing one or more services

where count < 1

Re: Events where a value is missing

jeremyhagand61 — Thu, 02 Jul 2020 23:36:41 GMT

I actually managed to solve this as follows:

source="WMI:Service" | dedup host Caption | transaction maxspan=60m host | table host,Caption | regex Caption!="Symantec Endpoint Protection"

Re: Events where a value is missing

jeremyhagand61 — Thu, 02 Jul 2020 23:59:52 GMT

I have tweaked this slightly to make it better.

source="WMI:Service" | transaction maxspan=1m host | dedup host,Caption | regex Caption!="Symantec Endpoint Protection" | table host,Caption

Re: Events where a value is missing

bowesmana — Fri, 03 Jul 2020 01:08:38 GMT

You are using the transaction command, which can have memory issues for long running transactions. From your solution, it looks like you're expecting Symantec Endpoint Protection to touch base every minute. An alternative approach avoiding the transaction command could be

source="WMI:Service" | bin _time span=1m | stats values(Caption) as Captions by _time host | where isnull(mvfind(Captions,"Symantec Endpoint Protection"))

which is asking to find all Captions for a host per minute, where there is no Symantec Endpoint Protection

it then will show you the Captions for that host within that minute for all minutes where there was no reqiured Caption.