Splunk Search

Sub Search failing due to return with no items

jmo1
Path Finder

I am very close but need some assistance.  I am attempting to create an alert based upon the criteria "Free Megabytes"<6000 AND "% Free Space" <10.  I have that logic working below...

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 | return 1000 host ]

 

The above code works as long as there is a hit for the Free Megabytes < 6000.  However if there are no hits, no host is returned to the % Free Space so it show all hosts that meet that critera.  How can this be adapted so that no hosts returned doesn't result in further query?  I am guessing eval, but my Splunk-fu is weak.

 

Any help is appreciated but actual code would be most helpful.

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try this on pre-8.1 Splunk

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 
  [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | appendpipe [ stats count | eval host="something that will never match" | where count==0 | fields - count ] ]

and try this on Splunk 8.1

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 
  [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | require ]
---
If this reply helps you, Karma would be appreciated.
0 Karma

jmo1
Path Finder

Thank you so much.  We are on 8.1, so I read up on the REQUIRE command.

I am seeing something very strange so maybe there is something else going on.

search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | appendpipe [ stats count | eval host="something that will never match" | where count==0 | fields - count ]

 

When I run the search above and when I run the 8.1 version with Require I get no returned items.  As I would expect.  However, if I change Value<6000 to Value<2000, I get incorrect results.  I don't understand how this can be because if it is less than 6000, it is also less than 2000.  Any ideas of what could be causing this? 

0 Karma

jmo1
Path Finder

Let me be more clear...

When I run...

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 
  [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | require ]

I get ZERO events, which is expected.

However, if I change the 6000 to a 2000, I get events returned.

 

To further diagnose, here is what I am seeing...

If I run the subsearch...

host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | require

I get servers returned.

If I change to 2000, no servers returned.

Therefore, it seems something in the first part is not handling correctly when no servers are returned.  

NOTE: I used the other (pre-8.1 logic) and it exhibits the exact same returns.

 

It appears with no returns from the subsearch, the first search is running against all servers.  I need the first search to fail if the subsearch returns nothing (which it seems is what the require command is suppost to do).

 

Thanks.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

"if it is less than 6000, it is also less than 2000" ? 5999 is less than 6000 but it isn't less than 2000!

0 Karma

jmo1
Path Finder

Did you even look at the code or what I am saying?  The query for less than 6000 returned nothing, but the query for less than 2000 retuned items.  Helpful responses are appreciated.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...