Splunk Search

Sub Search failing due to return with no items

jmo1
Path Finder

I am very close but need some assistance.  I am attempting to create an alert based upon the criteria "Free Megabytes"<6000 AND "% Free Space" <10.  I have that logic working below...

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 | return 1000 host ]

 

The above code works as long as there is a hit for the Free Megabytes < 6000.  However if there are no hits, no host is returned to the % Free Space so it show all hosts that meet that critera.  How can this be adapted so that no hosts returned doesn't result in further query?  I am guessing eval, but my Splunk-fu is weak.

 

Any help is appreciated but actual code would be most helpful.

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try this on pre-8.1 Splunk

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 
  [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | appendpipe [ stats count | eval host="something that will never match" | where count==0 | fields - count ] ]

and try this on Splunk 8.1

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 
  [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | require ]
---
If this reply helps you, Karma would be appreciated.
0 Karma

jmo1
Path Finder

Thank you so much.  We are on 8.1, so I read up on the REQUIRE command.

I am seeing something very strange so maybe there is something else going on.

search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | appendpipe [ stats count | eval host="something that will never match" | where count==0 | fields - count ]

 

When I run the search above and when I run the 8.1 version with Require I get no returned items.  As I would expect.  However, if I change Value<6000 to Value<2000, I get incorrect results.  I don't understand how this can be because if it is less than 6000, it is also less than 2000.  Any ideas of what could be causing this? 

0 Karma

jmo1
Path Finder

Let me be more clear...

When I run...

sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="% Free Space" Value<20 
  [ search host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | require ]

I get ZERO events, which is expected.

However, if I change the 6000 to a 2000, I get events returned.

 

To further diagnose, here is what I am seeing...

If I run the subsearch...

host=* sourcetype="Perfmon:Free Disk Space" instance!=HarddiskVolume* instance!=_Total counter="Free Megabytes" Value<6000 
  | return 1000 host 
  | require

I get servers returned.

If I change to 2000, no servers returned.

Therefore, it seems something in the first part is not handling correctly when no servers are returned.  

NOTE: I used the other (pre-8.1 logic) and it exhibits the exact same returns.

 

It appears with no returns from the subsearch, the first search is running against all servers.  I need the first search to fail if the subsearch returns nothing (which it seems is what the require command is suppost to do).

 

Thanks.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

"if it is less than 6000, it is also less than 2000" ? 5999 is less than 6000 but it isn't less than 2000!

0 Karma

jmo1
Path Finder

Did you even look at the code or what I am saying?  The query for less than 6000 returned nothing, but the query for less than 2000 retuned items.  Helpful responses are appreciated.

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...