How to plot multiple trendlines over multiple timechart values?

[esegal2u]

Hi,

I want to see multiple trendlines in a single graph, for multiple values.

My end-goal is to to have trendlines for each type of errors, so I can define an alert if any specific error had grown in comparison to the moving average count of that specific error. Something like (simplified):

sourcetype=x ERROR | timechart span=1m count as error_count | trendline sma10(error_count) as moving_avg_errors | eval spike=if(error_count > 2 * moving_avg_errors, 1000, 0)

This gives me a good output in total to all errors.
However, I want to see different errors separately.
When I do the following:

sourcetype=x ERROR | timechart span=1m count by error_msg limit=100

This gives me a timechart count of various error messages I have in my logs.
I would like to see independent trendlines for each of these messages.
Using "trendline sma10(?) as moving_avg_errors" obviously doesn't work this time, since it requires a specific field to work against.

How can I achieve this?
Am I looking at this search the wrong way?
Is it possible to make use of foreach in this case somehow?

Thanks for your help!

[jplumsdaine22]

There's a quicker way with streamstats that may be new - here's how to get a 7 hour rolling average when splitting by host for example 

<your base search>
| timechart span=1h sum(foo) as s by host
| untable _time host s
| sort host _time
| streamstats global=f window=7 avg(s) as trend by host
| timechart span=1h max(trend) by host

[drivernick]

I realize I'm a little late to this thread, but I had a similar issue and thought I might post my answer here for anyone running into the same problem.

sourcetype=x ERROR 
| timechart span=1m count by error_msg limit=100
| untable _time error count
| streamstats global=f window=10 avg(count) AS avg_count by error
| eval lowerBound=(-avg_count*2)
| eval upperBound=(avg_count*2)
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, "***".count."***", count)
| xyseries _time error isOutlier

The above code will show you all the data for each error over time, but it will surround outliers with asterisks. If you wanted to only show data when the count is an outlier, you could run the following:

sourcetype=x ERROR 
| timechart span=1m count by error_msg limit=100
| untable _time error count
| streamstats global=f window=10 avg(count) AS avg_count by error
| eval lowerBound=(-avg_count*2)
| eval upperBound=(avg_count*2)
| where count < lowerBound OR count > upperBound
| xyseries _time error count

Again, I know I'm late to the party, but hopefully this will help someone with a similar problem in the future.

[ksharma7]

@drivernick 

What if I have to count errors by grouping more than two fields like by error_msg and sitename say?

[jkat54]

I'm curious if trendline works with map:

sourcetype=x ERROR | timechart count by error_msg | map search='| trendline sma10($count$) as "moving_avg_errors".$error_msg$'

does this work for you?

[esegal2u]

Nope, doesn't work...

[jkat54]

Try sma10($count$)
AS $error_msg$

Instead of the field string concatenation in the above search.

[esegal2u]

It behaves the same. It matches events, but says "No results found".