How to find anomalies for requirement - Needed to detect the errors in Java ?

[venkatsm]

I would like to get the errors by class/exception/ExceptionMessage field (java based application errors) by week over week comparison. I checked timewrap but didn't fit in my requirement or I couldn't complete the query.

This query is based on ExceptionMessage in my application java log.

(index=MYINDEX1 OR index= MYINDEX2) level=ERROR earliest=-1h@h latest=now()  | rex field=message "(?<filter_error_msg>[^\{|\(|\@]+)" |stats count as current_hour by appName,filter_error_msg | appendcols [search (index=MYINDEX1 OR index= MYINDEX2) level=ERROR  earliest=-169h@h latest=-168h@h | rex field=message "(?<filter_error_msg>[^\{|\(|\@]+)" | stats count as last_week_same_hour by appName,filter_error_msg] |fillnull value=0 | where  ((current_hour > 1.2 * last_week_same_hour) AND current_hour > 25)

My goals are
1. Detect newly popped up errors and alert
2. If error rate increased compare to last week

I also saw anomalies command but not sure how to use it for this requirement. any suggestion on getting this splunk query. If I increase the timescale, the query completion is > 15mins which is very bad too. please suggest

Thanks

[deepashri_123]

Hi venkatsm,

I am not sure how much this will help but you can try using join and also fields like date_wday and date_hour for your comparison.
Also refer this blog for timewrap:
https://www.splunk.com/blog/2013/12/04/comparing-week-over-week-results.html
Let me know if this helps!!

[venkatsm]

Yes that doesn't help and I tried out. I have to put for 15 days timescale for 2 week data comparison. Search runtime increases because of increased timescale. I want to get my query to be completed <=15mins due to huge amount of data.

Thanks
venkat

[venkatsm]

How about 7.x Machine Learning concept, will it help in my use case ?.

Basically if splunk doesn't support, I have to do old school method. This method involves maintenance of script, local db, etc. If splunk provides without doing this method, it would be helpful.
1. Download the metrics from splunk by reading an API
2. Store locally in some mysql or some other db
3. Do slice and dice on data for week over week comparison
4. Send alert

[VS0909]

@venkatsm  Did you get the solution? Can you please share the solution with me as I am working on a similar problem.