Splunk Search

How do I find which item is missing from a static list?


We log job status messages in splunk.  When a job runs successfully, a success message is logged.  When a job errors out, an error message is logged.  Both types of messages include hostname as a field.  But when the underlying service fails to run a job, no message is logged.

I need to find hostnames that are missing success messages.  If I could use dataset literals, I might search something like this:

| FROM <list of expected hostnames as dataset literal> NOT [subsearch for success message hostnames]

But Splunk Cloud Platform apparently does not support the use of dataset literals, so I've resorted to a more convoluted process using stats, as suggested by several Internet authors:

<search for success message hostnames> | eval expected = split("<list of expected hostnames>"," ") | stats values(hostname) as hostname by expected | where NOT match (hostname,expected) 

This approach works if some, but not all, expected hostnames are missing.  However, in the case where all the expected hostnames are missing the search comes back empty.  I understand why it comes back empty.  What I need is a "correct" way to find these missing hostnames that will work in all cases.

Labels (3)
0 Karma


I may have solved this.

The search

| makeresults | eval <fieldname> = split(<list as string>,<delimiter>) | mvexpand <fieldname> | search ...

generates the static "dataset literal" I need, to which I can append the NOT condition to filter out the log entries seen, leaving behind the ones that are missing.  Would still appreciate any better solutions you might have for this.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...