Splunk Search

How to get pattern recognition in Splunk

MousumiChowdhur
Contributor

Hi,

I want to get my event patterns to be recognized automatically. The pattern is not uniform but Splunk should identify any small difference in the events and should give the trend or count of the patterns over time. How can I achieve this?

0 Karma

saurabh_tek11
Communicator

There is a very simple way of doing this - In your event, there is a default field called punct.

This seem like some alien language which is not understandable at the first look. But its very helpful one. How it works is - in a event, it strips all letters, numbers and replace Whitespace with the Underscore. Leaving just the PUNCTuation.

Best part is this field gets extracted by Splunk automatically.

We can directly separate a specific type of events belonging to to specific pattern. We use the punct field to find anomalies in data.

For example, If 99% of your events are like this ____::__[]:________...___ and 1% look like this ..._-_-_[//:::]_"_//.?=__."___"://../.?=&=-"_"/._( then we can easily find the odd one out (undesired one) using this field.

This will show the count of patterns among your events. All events of same patterns will be grouped.

It is a fantastic way to quickly point you to the outliers that didn't match the pattern you expected.

Very helpful in finding anomalous event among large data set OR writing complex regex's for field extraction to ensure all events are covered.

more information about punct is here. I hope this answers your question. 🙂 Thank you - Saurabh

saurabh_tek11
Communicator
0 Karma

saurabh_tek11
Communicator

@MousumiChowdhury - does this helps ?

0 Karma

saurabh_tek11
Communicator

@MousumiChowdhury - hope this answers your question as this way you dont have to write a custom search and you can use a default fields to get the pattern matching. if it supports your question, please accept this answer.

0 Karma

MousumiChowdhur
Contributor

I have used the below query to find the pattern recognition which is working fine for me:

index=<index> | cluster t=0.7 labelonly=t | findkeywords labelfield=cluster_label | table sampleEvent percentInInputGroup | sort - percentInInputGroup

ddrillic
Ultra Champion

Maybe the following would be useful - Detecting patterns

0 Karma

MousumiChowdhur
Contributor

Hi!

I have tried using cluster. Below is my query:

index=<index> | cluster showcount=t t=0.7 labelonly=t | table _time cluster_count cluster_label _raw | dedup 1 cluster_label | sort - cluster_count cluster_label _time | chart values(cluster_count) as count by _raw | sort limit=20 - count

Is this a correct approach to find the latest patterns that have occurred the most?

0 Karma

indeed_2000
Motivator

@MousumiChowdhur Thanks it work, but some lines are huge specially exception one, how can trim only first line of error?

e.g. current output

2022-04-25 15:35:10,514 ERROR [APP] User User1 invalid: javax.security.auth.login.LoginException: User T75171 invalid at ws.loginmodule.Spi.login(LoginModuleSpi.java:356) [loginModule2-1.0.0-SNAPSHOT.jar:] at ws.loginmodule.ModuleSpi.login(LoginModuleSpi.java:172) [loginModule2-1.0.0-SNAPSHOT.jar:] at sun.reflect.GeneratedMethodAccessor1495.invoke(Unknown Source) [:1.8.0_275] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_275] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_275] at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) [rt.jar:1.8.0_275] at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) [rt.jar:1.8.0_275] at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) [rt.jar:1.8.0_275]

...

 

expected output:

_raw                                                                                                                                                                                                                                          count 

2022-04-25 15:35:10,514 ERROR [APP] User User1 invalid: javax.security.auth.login.LoginException        550    

any idea?

Thanks

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...