Alerting

Alterting on value returned by the Splunk add-on for Unix and Linux

davidcaughill
Explorer

Forgive my ignorance as I'm relatively new to Splunk. I'm currently hitting what I *think* is a data type issue, but I'm not quite sure how to proceed. We are using the Splunk add-on for Unix and Linux to return the set of 'df-metric' values. I would like to set up a simple alert on the metric_name:df_metric.UsePct value, alerting when the value exceeds 85%. I'm able to run this query and return data using an equality operator on that value: 

 

 

 

index="linuxlogs" sourcetype="df_metric" host="ip-xxx-xx-xx-x" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=8

 

 

 

 

...however I'm NOT able to return data when perform an 'greater than' comparison on the metric_name:df_metric.UsePct value like this: 

 

 

 

index="linuxlogs" sourcetype="df_metric" host="ip-xxx-xx-xx-x" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct">8

 

 

 

Initially I tried manipulating the metric_name:df_metric.UsePct with the tonumber() function, thinking I was possibly receiving a string back, however that does not result in the data I would expect to see. 

If anyone has guidance on traversing the data set returned by df_metric or any other points, I would appreciate it! 

Thank you! 

NOTE: I'm using  as a value for the metric_name:df_metric.UsePct only for testing purposes. This will, of course, need to be adjusted to 85 for the live alert. 

Labels (1)
0 Karma
1 Solution

renjith_nair
SplunkTrust
SplunkTrust

Alright!

It looks like the pct value is a multi value field for some reason. Probably the values are of metrics and you are not using a metrics index.

You can confirm this by

index= sourcetype="df_metric"
|eval mc=mvcount('metric_name:df_metric.UsePct')
|table metric_name*,mc

Nevertheless,just try this and see if it produces any result

your search
|eval df_pct=mvindex('metric_name:df_metric.UsePct',0)
|where df_pct > 8

 

Happy Splunking!

View solution in original post

renjith_nair
SplunkTrust
SplunkTrust

Try using where or search

e.g.

 

index="linuxlogs" sourcetype="df_metric" host="ip-xxx-xx-xx-x" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=*|search "metric_name:df_metric.UsePct" > 8

 

 

Happy Splunking!

davidcaughill
Explorer

Thank you and yes, that makes sense to me as well, however that yields a type error: 

Error in 'where' command: Type checking failed. The '>' operator received different types.

 

davidcaughill_0-1607542397928.png

 

I also tried wrapping the 'df_metric.UsePct' value in the 'where' clause in a  'tonumber()' call to force a type conversion, however that does not seem to yield anything. 

 

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Yes, sorry I didnt pay attention to the fieldname. There error is because, we are comparing a "string" with a number. As mention, you may 'search' instead - I updated the answer

So two options

"your current search" |search "metric_name:df_metric.UsePct" > 8

or

"your current search"|rename "metric_name:df_metric.UsePct" as df_pct |where df_pct > 8
Happy Splunking!

davidcaughill
Explorer

It appears both 'search' and 'where' result in a string type. For example, when I perform this search: 

index="linuxlogs" sourcetype="df_metric" host="<redacted>"  Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=* | search "metric_name:df_metric.UsePct" = 8

 

I am able to retrieve results, I assume because the search "metric_name:df_metrci.UsePct" = 8 is correctly performing an equality test on the 8 as a string value:

davidcaughill_0-1607611631886.png

 

However when I change this to the ">" operator and value, it does not return the result above, which should meet this search criteria:

index="linuxlogs" sourcetype="df_metric" host="<redacted>"  Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=* | search "metric_name:df_metric.UsePct" > 0

 

Do you have any suggestions to on how to force a type conversion on the value being returned in "metric_name:df_metric.UsePct"? 

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Alright!

It looks like the pct value is a multi value field for some reason. Probably the values are of metrics and you are not using a metrics index.

You can confirm this by

index= sourcetype="df_metric"
|eval mc=mvcount('metric_name:df_metric.UsePct')
|table metric_name*,mc

Nevertheless,just try this and see if it produces any result

your search
|eval df_pct=mvindex('metric_name:df_metric.UsePct',0)
|where df_pct > 8

 

Happy Splunking!

davidcaughill
Explorer

Perfect! You are right - it is an index and that solution worked. Thank you very much!

In hindsight, the UI was trying to tell me that with the multiple values appearing in the 'df_metric.UsePct' field in the screenshot I posted above...I should have seen that. 

I'm saving both of these queries off for future use - thank you again @renjith_nair ! 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...