Dashboards & Visualizations

Search query to extract the field from the raw data for different patterns

aditsss
Builder

Hi Everyone,

Below are my logs :

2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:PULL_GRS_FILE_UPLOAD

2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:DOWNLOAD_S3

2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:UPLOAD_S3

In the end of each logs I have some pattern PULL_GRS_FILE_UPLOAD,DOWNLOAD_S3 and UPLOAD_S3 . 

I want to display these patterns with their counts.

Can someone guide me with the search query for this.

As of now I am seeing the events by using below search:

index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3"

Labels (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3" OR "Invoked:PULL_GRS_FILE_UPLOAD" OR "Invoked:UPLOAD_S3"
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

Add the other patterns in a similar manner

View solution in original post

gcusello
Legend

Hi @aditsss,

your have to extract the values as a field using the rex command and then use this field for the stats count command, something like this:

index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3"
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

You can test the regex at https://regex101.com/r/NGX1Yy/1

Ciao.

Giuseppe

aditsss
Builder

@gcusello 

Below search query is extracting one "Invoked:DOWNLOAD_S3 " but there are other patterns also present in logs Invoked:UPLOAD_S3 and Invoked:PULL_GRS_FILE_UPLOAD .

Total there are seven patterns . I want only one search query to extract all patterns .

What changes I should make in below search query to include all patterns.

index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3"
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3" OR "Invoked:PULL_GRS_FILE_UPLOAD" OR "Invoked:UPLOAD_S3"
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

Add the other patterns in a similar manner

View solution in original post

gcusello
Legend

Hi @aditsss,

good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

0 Karma

inventsekar
Super Champion

Question - Below search query is extracting one "Invoked:DOWNLOAD_S3 " but there are other patterns also present in logs Invoked:UPLOAD_S3 and Invoked:PULL_GRS_FILE_UPLOAD .

Total there are seven patterns . I want only one search query to extract all patterns.

 

answer - 

index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3"

 on your search query, you are filtering one pattern. if you dont filter, then, the rex will pickup all patterns.

>>> Happy Splunking !

inventsekar
Super Champion
| makeresults 
| eval msg="2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:PULL_GRS_FILE_UPLOAD" 
| append 
    [| makeresults 
    | eval msg="2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:DOWNLOAD_S3" ]
    | rex field=msg ":\s+(?<pattern>.*)$" 
    | table pattern

Hi @aditsss i included 2 logs as msg and the rex is picking up the pattern fine. so, once you picked up the pattern fine, you can do the count easily. 

makeresults-multi-events.jpg

please let us know if you got some more queries.. thanks. 

 

(PS - i have given around 500+ karma points so far, received badge for that, if an answer helped you, a karma point would be nice!. we all should start "Learn, Give Back, Have Fun")

>>> Happy Splunking !

gcusello
Legend

Hi @aditsss,

I used your search as main search, you can modify the main search and use my approiach to extract field and use it in stats.

index=abc ns=xyz app_name=ok
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

Ciao.

Giuseppe

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!