Alterting on value returned by the Splunk add-on for Unix and Linux

[davidcaughill]

Forgive my ignorance as I'm relatively new to Splunk. I'm currently hitting what I *think* is a data type issue, but I'm not quite sure how to proceed. We are using the Splunk add-on for Unix and Linux to return the set of 'df-metric' values. I would like to set up a simple alert on the metric_name:df_metric.UsePct value, alerting when the value exceeds 85%. I'm able to run this query and return data using an equality operator on that value: 

index="linuxlogs" sourcetype="df_metric" host="ip-xxx-xx-xx-x" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=8

...however I'm NOT able to return data when perform an 'greater than' comparison on the metric_name:df_metric.UsePct value like this: 

index="linuxlogs" sourcetype="df_metric" host="ip-xxx-xx-xx-x" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct">8

Initially I tried manipulating the metric_name:df_metric.UsePct with the tonumber() function, thinking I was possibly receiving a string back, however that does not result in the data I would expect to see. 

If anyone has guidance on traversing the data set returned by df_metric or any other points, I would appreciate it! 

Thank you! 

NOTE: I'm using 8  as a value for the metric_name:df_metric.UsePct only for testing purposes. This will, of course, need to be adjusted to 85 for the live alert. 

[renjith_nair]

Try using where or search

e.g.

index="linuxlogs" sourcetype="df_metric" host="ip-xxx-xx-xx-x" Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=*|search "metric_name:df_metric.UsePct" > 8

[davidcaughill]

Thank you and yes, that makes sense to me as well, however that yields a type error: 

Error in 'where' command: Type checking failed. The '>' operator received different types.

I also tried wrapping the 'df_metric.UsePct' value in the 'where' clause in a  'tonumber()' call to force a type conversion, however that does not seem to yield anything. 

[renjith_nair]

Yes, sorry I didnt pay attention to the fieldname. There error is because, we are comparing a "string" with a number. As mention, you may 'search' instead - I updated the answer

So two options

"your current search" |search "metric_name:df_metric.UsePct" > 8

or

"your current search"|rename "metric_name:df_metric.UsePct" as df_pct |where df_pct > 8

[davidcaughill]

It appears both 'search' and 'where' result in a string type. For example, when I perform this search: 

index="linuxlogs" sourcetype="df_metric" host="<redacted>"  Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=* | search "metric_name:df_metric.UsePct" = 8

I am able to retrieve results, I assume because the search "metric_name:df_metrci.UsePct" = 8 is correctly performing an equality test on the 8 as a string value:

However when I change this to the ">" operator and value, it does not return the result above, which should meet this search criteria:

index="linuxlogs" sourcetype="df_metric" host="<redacted>"  Filesystem = "/dev/xvda1" "metric_name:df_metric.UsePct"=* | search "metric_name:df_metric.UsePct" > 0

Do you have any suggestions to on how to force a type conversion on the value being returned in "metric_name:df_metric.UsePct"? 

[renjith_nair · Accepted solution]

Alright!

It looks like the pct value is a multi value field for some reason. Probably the values are of metrics and you are not using a metrics index.

You can confirm this by

index= sourcetype="df_metric"
|eval mc=mvcount('metric_name:df_metric.UsePct')
|table metric_name*,mc

Nevertheless,just try this and see if it produces any result

your search
|eval df_pct=mvindex('metric_name:df_metric.UsePct',0)
|where df_pct > 8

[davidcaughill]

Perfect! You are right - it is an index and that solution worked. Thank you very much!

In hindsight, the UI was trying to tell me that with the multiple values appearing in the 'df_metric.UsePct' field in the screenshot I posted above...I should have seen that. 

I'm saving both of these queries off for future use - thank you again @renjith_nair !