topic Re: How to get pattern recognition in Splunk in Splunk Search

How to get pattern recognition in Splunk

MousumiChowdhur — Mon, 18 Dec 2017 09:38:55 GMT

Hi,

I want to get my event patterns to be recognized automatically. The pattern is not uniform but Splunk should identify any small difference in the events and should give the trend or count of the patterns over time. How can I achieve this?

Re: How to get pattern recognition in Splunk

ddrillic — Mon, 18 Dec 2017 10:55:58 GMT

Maybe the following would be useful - Detecting patterns

Re: How to get pattern recognition in Splunk

MousumiChowdhur — Mon, 18 Dec 2017 12:16:25 GMT

Hi!

I have tried using cluster. Below is my query:

index=<index> | cluster showcount=t t=0.7 labelonly=t | table _time cluster_count cluster_label _raw | dedup 1 cluster_label | sort - cluster_count cluster_label _time | chart values(cluster_count) as count by _raw | sort limit=20 - count

Is this a correct approach to find the latest patterns that have occurred the most?

Re: How to get pattern recognition in Splunk

MousumiChowdhur — Wed, 20 Dec 2017 06:42:11 GMT

I have used the below query to find the pattern recognition which is working fine for me:

index=<index> | cluster t=0.7 labelonly=t | findkeywords labelfield=cluster_label | table sampleEvent percentInInputGroup | sort - percentInInputGroup

Re: How to get pattern recognition in Splunk

saurabh_tek11 — Wed, 20 Dec 2017 08:15:47 GMT

There is a very simple way of doing this - In your event, there is a default field called punct.

This seem like some alien language which is not understandable at the first look. But its very helpful one. How it works is - in a event, it strips all letters, numbers and replace Whitespace with the Underscore. Leaving just the PUNCTuation.

Best part is this field gets extracted by Splunk automatically.

We can directly separate a specific type of events belonging to to specific pattern. We use the punct field to find anomalies in data.

For example, If 99% of your events are like this ____::__[]:________...___ and 1% look like this ..._-_-_[//:::]_"_//.?=__."___"://../.?=&=-"_"/._( then we can easily find the odd one out (undesired one) using this field.

This will show the count of patterns among your events. All events of same patterns will be grouped.

It is a fantastic way to quickly point you to the outliers that didn't match the pattern you expected.

Very helpful in finding anomalous event among large data set OR writing complex regex's for field extraction to ensure all events are covered.

more information about punct is here. I hope this answers your question. 🙂 Thank you - Saurabh

Re: How to get pattern recognition in Splunk

saurabh_tek11 — Wed, 20 Dec 2017 08:42:12 GMT

As in Splunk 6.3, Use the punct field to search on similar events

Re: How to get pattern recognition in Splunk

saurabh_tek11 — Thu, 21 Dec 2017 10:25:35 GMT

@MousumiChowdhury - does this helps ?

Re: How to get pattern recognition in Splunk

saurabh_tek11 — Wed, 03 Jan 2018 09:41:26 GMT

@MousumiChowdhury - hope this answers your question as this way you dont have to write a custom search and you can use a default fields to get the pattern matching. if it supports your question, please accept this answer.

Re: How to get pattern recognition in Splunk

indeed_2000 — Tue, 26 Apr 2022 04:57:52 GMT

@MousumiChowdhur Thanks it work, but some lines are huge specially exception one, how can trim only first line of error?

e.g. current output

2022-04-25 15:35:10,514 ERROR [APP] User User1 invalid: javax.security.auth.login.LoginException: User T75171 invalid at ws.loginmodule.Spi.login(LoginModuleSpi.java:356) [loginModule2-1.0.0-SNAPSHOT.jar:] at ws.loginmodule.ModuleSpi.login(LoginModuleSpi.java:172) [loginModule2-1.0.0-SNAPSHOT.jar:] at sun.reflect.GeneratedMethodAccessor1495.invoke(Unknown Source) [:1.8.0_275] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_275] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_275] at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) [rt.jar:1.8.0_275] at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) [rt.jar:1.8.0_275] at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) [rt.jar:1.8.0_275]

...

expected output:

_raw count

2022-04-25 15:35:10,514 ERROR [APP] User User1 invalid: javax.security.auth.login.LoginException 550

any idea?

Thanks