Dashboards & Visualizations

How to plot multiple trendlines over multiple timechart values?

esegal2u
Explorer

Hi,

I want to see multiple trendlines in a single graph, for multiple values.

My end-goal is to to have trendlines for each type of errors, so I can define an alert if any specific error had grown in comparison to the moving average count of that specific error. Something like (simplified):

sourcetype=x ERROR | timechart span=1m count as error_count | trendline sma10(error_count) as moving_avg_errors | eval spike=if(error_count > 2 * moving_avg_errors, 1000, 0)

This gives me a good output in total to all errors.
However, I want to see different errors separately.
When I do the following:

sourcetype=x ERROR | timechart span=1m count by error_msg limit=100

This gives me a timechart count of various error messages I have in my logs.
I would like to see independent trendlines for each of these messages.
Using "trendline sma10(?) as moving_avg_errors" obviously doesn't work this time, since it requires a specific field to work against.

How can I achieve this?
Am I looking at this search the wrong way?
Is it possible to make use of foreach in this case somehow?

Thanks for your help!

Tags (1)

jplumsdaine22
Influencer

There's a quicker way with streamstats that may be new - here's how to get a 7 hour rolling average when splitting by host for example 

 

<your base search>
| timechart span=1h sum(foo) as s by host
| untable _time host s
| sort host _time
| streamstats global=f window=7 avg(s) as trend by host
| timechart span=1h max(trend) by host

 

drivernick
Engager

I realize I'm a little late to this thread, but I had a similar issue and thought I might post my answer here for anyone running into the same problem.

sourcetype=x ERROR 
| timechart span=1m count by error_msg limit=100
| untable _time error count
| streamstats global=f window=10 avg(count) AS avg_count by error
| eval lowerBound=(-avg_count*2)
| eval upperBound=(avg_count*2)
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, "***".count."***", count)
| xyseries _time error isOutlier

The above code will show you all the data for each error over time, but it will surround outliers with asterisks. If you wanted to only show data when the count is an outlier, you could run the following:

sourcetype=x ERROR 
| timechart span=1m count by error_msg limit=100
| untable _time error count
| streamstats global=f window=10 avg(count) AS avg_count by error
| eval lowerBound=(-avg_count*2)
| eval upperBound=(avg_count*2)
| where count < lowerBound OR count > upperBound
| xyseries _time error count

Again, I know I'm late to the party, but hopefully this will help someone with a similar problem in the future.

ksharma7
Path Finder

@drivernick 

What if I have to count errors by grouping more than two fields like by error_msg and sitename say?

 

0 Karma

jkat54
SplunkTrust
SplunkTrust

I'm curious if trendline works with map:

sourcetype=x ERROR | timechart count by error_msg | map search='| trendline sma10($count$) as "moving_avg_errors".$error_msg$'

does this work for you?

0 Karma

esegal2u
Explorer

Nope, doesn't work...

0 Karma

jkat54
SplunkTrust
SplunkTrust

Try sma10($count$)
AS $error_msg$

Instead of the field string concatenation in the above search.

0 Karma

esegal2u
Explorer

It behaves the same. It matches events, but says "No results found".

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...