Splunk Search

Finding when difference between servers greater than 50%

Adacats
Engager

I am using the below query (server names replaced) to find when there is a greater than 50% difference in volume between 2 call routers (servers). For some reason im getting no timechart results, even when setting the difference to 1% which should always return results.

index=OMITTED source=OMITTED host="SERVER1" OR host="SERVER2"
| stats max(Value) as Value by host
| eventstats max(if(host='SERVER1', Value, null)) as server1_value max(if(host='SERVER2', Value, null)) as server2_value
| eval value_difference = abs(server1_value - server2_value)
| eval value_percentage_difference = if(coalesce(server1_value, server2_value) != 0, (value_difference / coalesce(server1_value, server2_value) * 100), 0)
| where value_percentage_difference > 1
| timechart avg(value_percentage_difference)
Labels (2)
0 Karma

yuanliu
SplunkTrust
SplunkTrust

Note you need to place source=OMITTED host="SERVER1" OR host="SERVER2" in parentheses; alternatively use IN operator.  Finding difference should not be that complicated.

index=_internal earliest=-15mindex=OMITTED source=OMITTED host IN ("SERVER1", "SERVER2")
| stats max(Value) as Value by host
| stats max(Value) as max_of_two min(Value) as min_of_two
| where max_of_two / min_of_two > 0.75 

However, your OP says you want timechart.  That's why @richgalloway includes _time in groupby in that first stats.  But you can substitute the first stats with timechart to simplify this, then use the same technique in every row to find percent deviation.

index=_internal earliest=-15mindex=OMITTED source=OMITTED host IN ("SERVER1", "SERVER2")
| timechart span=1d max(Value) as Value by host
| eventstats max(Value) as max_of_two min(Value) as min_of_two
| where max_of_two / min_of_two > 0.75 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The query returns no results because the timechart command requires the _time field, but that field was removed by the stats command on line 2.

The fix is to include _time in the stats command, like this

index=OMITTED source=OMITTED host="SERVER1" OR host="SERVER2"
| bin span=1d _time
| stats max(Value) as Value by host, _time
| eventstats ...
| timechart span=1d avg(value_percentage_difference)

 Adjust the span option in the bin and timechart commands to preference.  Make sure they match.

---
If this reply helps you, Karma would be appreciated.

Adacats
Engager

hmm i might be doing something wrong still as i get the timechart but the results are all zeros and there should be a couple at least above zero

 

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

How to find the worst searches in your Splunk environment and how to fix them

Everyone knows Splunk is a powerful platform for running searches and doing data analytics. Your ...

Share Your Feedback: On Admin Config Service (ACS)!

Help Us Build a Better Admin Config Service Experience (ACS)   We Want Your Feedback on Admin Config Service ...