Dashboards & Visualizations

Search query to extract the field from the raw data for different patterns

aditsss
Motivator

Hi Everyone,

Below are my logs :

2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:PULL_GRS_FILE_UPLOAD

2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:DOWNLOAD_S3

2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:UPLOAD_S3

In the end of each logs I have some pattern PULL_GRS_FILE_UPLOAD,DOWNLOAD_S3 and UPLOAD_S3 . 

I want to display these patterns with their counts.

Can someone guide me with the search query for this.

As of now I am seeing the events by using below search:

index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3"

Labels (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3" OR "Invoked:PULL_GRS_FILE_UPLOAD" OR "Invoked:UPLOAD_S3"
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

Add the other patterns in a similar manner

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @aditsss,

your have to extract the values as a field using the rex command and then use this field for the stats count command, something like this:

index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3"
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

You can test the regex at https://regex101.com/r/NGX1Yy/1

Ciao.

Giuseppe

aditsss
Motivator

@gcusello 

Below search query is extracting one "Invoked:DOWNLOAD_S3 " but there are other patterns also present in logs Invoked:UPLOAD_S3 and Invoked:PULL_GRS_FILE_UPLOAD .

Total there are seven patterns . I want only one search query to extract all patterns .

What changes I should make in below search query to include all patterns.

index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3"
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3" OR "Invoked:PULL_GRS_FILE_UPLOAD" OR "Invoked:UPLOAD_S3"
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

Add the other patterns in a similar manner

gcusello
SplunkTrust
SplunkTrust

Hi @aditsss,

good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

0 Karma

inventsekar
Ultra Champion

Question - Below search query is extracting one "Invoked:DOWNLOAD_S3 " but there are other patterns also present in logs Invoked:UPLOAD_S3 and Invoked:PULL_GRS_FILE_UPLOAD .

Total there are seven patterns . I want only one search query to extract all patterns.

 

answer - 

index=abc ns=xyz app_name=ok "Invoked:DOWNLOAD_S3"

 on your search query, you are filtering one pattern. if you dont filter, then, the rex will pickup all patterns.

inventsekar
Ultra Champion
| makeresults 
| eval msg="2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:PULL_GRS_FILE_UPLOAD" 
| append 
    [| makeresults 
    | eval msg="2020-10-05T09:12:25.457507609Z app_name=abc environment=e2 ns=c2_container=arc-api pod_name=deployment-51-csl4p message=2020-10-05 02:12:25.456 INFO [arc-service,3b5bbd7422319fde,3b5bbd7422319fde,true] 1 --- [or-http-epoll-4] c.a.b.arc.controller.ARCFileController : Invoked:DOWNLOAD_S3" ]
    | rex field=msg ":\s+(?<pattern>.*)$" 
    | table pattern

Hi @aditsss i included 2 logs as msg and the rex is picking up the pattern fine. so, once you picked up the pattern fine, you can do the count easily. 

makeresults-multi-events.jpg

please let us know if you got some more queries.. thanks. 

 

(PS - i have given around 500+ karma points so far, received badge for that, if an answer helped you, a karma point would be nice!. we all should start "Learn, Give Back, Have Fun")

gcusello
SplunkTrust
SplunkTrust

Hi @aditsss,

I used your search as main search, you can modify the main search and use my approiach to extract field and use it in stats.

index=abc ns=xyz app_name=ok
| rex ":\s+(?<pattern>.*)$"
| stats count BY pattern

Ciao.

Giuseppe

Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...