Splunk Search

How do I find which item is missing from a static list?

Bennette
Explorer

We log job status messages in splunk.  When a job runs successfully, a success message is logged.  When a job errors out, an error message is logged.  Both types of messages include hostname as a field.  But when the underlying service fails to run a job, no message is logged.

I need to find hostnames that are missing success messages.  If I could use dataset literals, I might search something like this:

| FROM <list of expected hostnames as dataset literal> NOT [subsearch for success message hostnames]

But Splunk Cloud Platform apparently does not support the use of dataset literals, so I've resorted to a more convoluted process using stats, as suggested by several Internet authors:

<search for success message hostnames> | eval expected = split("<list of expected hostnames>"," ") | stats values(hostname) as hostname by expected | where NOT match (hostname,expected) 

This approach works if some, but not all, expected hostnames are missing.  However, in the case where all the expected hostnames are missing the search comes back empty.  I understand why it comes back empty.  What I need is a "correct" way to find these missing hostnames that will work in all cases.

Labels (2)
0 Karma

Bennette
Explorer

I may have solved this.

The search

| makeresults | eval <fieldname> = split(<list as string>,<delimiter>) | mvexpand <fieldname> | search ...

generates the static "dataset literal" I need, to which I can append the NOT condition to filter out the log entries seen, leaving behind the ones that are missing.  Would still appreciate any better solutions you might have for this.

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...