Splunk Search
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question
Solved! Jump to solution

Bring several events together.

michaelteck
Explorer
‎03-26-2024 02:55 AM

Hello everyone, 

I'm coming to you for advice. I am currently working with splunk to create monitor WSO2-APIM instances. 
According to the WSO2-APIM documentation, logs are generated as follows : 

[2019-12-12 17:30:08,091] DEBUG - wire HTTPS-Listener I/O dispatcher-5 >> "GET /helloWorld/1.0.0 HTTP/1.1[\r][\n]"
[2019-12-12 17:30:08,093] DEBUG - wire HTTPS-Listener I/O dispatcher-5 >> "Host: localhost:8243[\r][\n]"
[2019-12-12 17:30:08,094] DEBUG - wire HTTPS-Listener I/O dispatcher-5 >> "User-Agent: curl/7.54.0[\r][\n]"
[2019-12-12 17:30:08,095] DEBUG - wire HTTPS-Listener I/O dispatcher-5 >> "accept: */*[\r][\n]"
[2019-12-12 17:30:08,096] DEBUG - wire HTTPS-Listener I/O dispatcher-5 >> "Authorization: Bearer 07f6b26d-0f8d-312a-8d38-797e054566cd[\r][\n]"
[2019-12-12 17:30:08,097] DEBUG - wire HTTPS-Listener I/O dispatcher-5 >> "[\r][\n]"
[2019-12-12 17:30:08,105] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "GET /v2/5df22aa131000084009a30a9 HTTP/1.1[\r][\n]"
[2019-12-12 17:30:08,106] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "accept: */*[\r][\n]"
[2019-12-12 17:30:08,107] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "Host: www.mocky.io[\r][\n]"
[2019-12-12 17:30:08,108] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "Connection: Keep-Alive[\r][\n]"
[2019-12-12 17:30:08,109] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "User-Agent: Synapse-PT-HttpComponents-NIO[\r][\n]"
[2019-12-12 17:30:08,110] DEBUG - wire HTTP-Sender I/O dispatcher-1 << "[\r][\n]"
[2019-12-12 17:30:08,266] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "HTTP/1.1 200 OK[\r][\n]"
[2019-12-12 17:30:08,268] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "Server: Cowboy[\r][\n]"
[2019-12-12 17:30:08,269] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "Connection: keep-alive[\r][\n]"
[2019-12-12 17:30:08,271] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "Date: Thu, 12 Dec 2019 12:00:08 GMT[\r][\n]"
[2019-12-12 17:30:08,272] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "Content-Type: application/json[\r][\n]"
[2019-12-12 17:30:08,273] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "Content-Length: 20[\r][\n]"
[2019-12-12 17:30:08,274] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "Via: 1.1 vegur[\r][\n]"
[2019-12-12 17:30:08,275] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "[\r][\n]"
[2019-12-12 17:30:08,276] DEBUG - wire HTTP-Sender I/O dispatcher-1 >> "{ "hello": "world" }"
[2019-12-12 17:30:08,282] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "HTTP/1.1 200 OK[\r][\n]"
[2019-12-12 17:30:08,283] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "Access-Control-Expose-Headers: [\r][\n]"
[2019-12-12 17:30:08,284] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "Access-Control-Allow-Origin: *[\r][\n]"
[2019-12-12 17:30:08,285] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "Access-Control-Allow-Methods: GET[\r][\n]"
[2019-12-12 17:30:08,286] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "Access-Control-Allow-Headers: authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Authorization[\r][\n]"
[2019-12-12 17:30:08,287] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "Content-Type: application/json[\r][\n]"
[2019-12-12 17:30:08,287] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "Via: 1.1 vegur[\r][\n]"
[2019-12-12 17:30:08,288] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "Date: Thu, 12 Dec 2019 12:00:08 GMT[\r][\n]"
[2019-12-12 17:30:08,289] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "Transfer-Encoding: chunked[\r][\n]"
[2019-12-12 17:30:08,290] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "[\r][\n]"
[2019-12-12 17:30:08,290] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "14[\r][\n]"
[2019-12-12 17:30:08,291] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "{ "hello": "world" }[\r][\n]"
[2019-12-12 17:30:08,292] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "0[\r][\n]"
[2019-12-12 17:30:08,293] DEBUG - wire HTTPS-Listener I/O dispatcher-5 << "[\r][\n]"


And also according to the doc : 

DEBUG - wire >>	Represents the message coming into the API Gateway from the wire.
DEBUG - wire <<	Represents the message that goes to the wire from the API Gateway.

 

I use AWS Lambda to retrieve the WSO2-APIM logs, which are stored in AWS CloudWatch.
I've just started using Splunk so I'm not very good at SPL.
I would like Splunk to process events with SPL and then output something like this :

Date, loglevel, action_https, correlationID, message, duration
[2019-12-12 17:30:08,091], DEBUG, HTTPS-Listener, dispatcher-5, "GET /helloWorld/1.0.0 HTTP/1.1[\r][\n]" "Host: localhost:8243[\r][\n]" "User-Agent: curl/7.54.0[\r][\n]" "accept: */*[\r][\n]" "Authorization: Bearer 07f6b26d-0f8d-312a-8d38-797e054566cd[\r][\n]" "[\r][\n]", 006
[2019-12-12 17:30:08,105], DEBUG, HTTPS-Listener, dispatcher-1, "GET /v2/5df22aa131000084009a30a9 HTTP/1.1[\r][\n]" "accept: */*[\r][\n]" "Host: www.mocky.io[\r][\n]" "Connection: Keep-Alive[\r][\n]" "User-Agent: Synapse-PT-HttpComponents-NIO[\r][\n]" "[\r][\n]", 005
[2019-12-12 17:30:08,266], DEBUG, HTTPS-Sender, dispatcher-1, "HTTP/1.1 200 OK[\r][\n]" "Server: Cowboy[\r][\n]" "Connection: keep-alive[\r][\n]" "Date: Thu, 12 Dec 2019 12:00:08 GMT[\r][\n]" "Content-Type: application/json[\r][\n]" "Content-Length: 20[\r][\n]" "Via: 1.1 vegur[\r][\n]" "[\r][\n]" "{ "hello": "world" }", 010
[2019-12-12 17:30:08,282], DEBUG, HTTPS-Listener, dispatcher-5, "HTTP/1.1 200 OK[\r][\n]" "Access-Control-Expose-Headers: [\r][\n]" "Access-Control-Allow-Origin: *[\r][\n]" "Access-Control-Allow-Methods: GET[\r][\n]" "Access-Control-Allow-Headers: authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Authorization[\r][\n]" "Content-Type: application/json[\r][\n]" "Via: 1.1 vegur[\r][\n]" "Date: Thu, 12 Dec 2019 12:00:08 GMT[\r][\n]" "Transfer-Encoding: chunked[\r][\n]" "[\r][\n]" "14[\r][\n]" "{ "hello": "world" }[\r][\n]" "0[\r][\n]" "[\r][\n]", 011


Do you have any ideas on how to do this with SPL in the Search App?

Thank you for those who took the time to read and reply to me.

Labels (1)
Labels
Tags (3)
0 Karma
Reply
1 Solution
Solved! Jump to solution
Solution

ITWhisperer
SplunkTrust
SplunkTrust
‎03-26-2024 07:56 AM

Try something like this

``` Reverse the order of events so that earliest is first ```
| reverse
``` Extract the fields ```
| rex "(?<Date>\[[^\]]+\])\s(?<loglevel>\w+)\s-\swire\s(?<action_https>\S+)\sI\/O\s(?<correlationID>\S+)\s(?<direction>\S+)\s(?<message>.*)"
``` Tag the events in order to be able to maintain the sequence ```
| streamstats count as event
``` Create a direction-based grouping for correlatoinIds ```
| eval grouping=correlationID.direction
``` Sort so that events for the same correlationID are together and in sequence ```
| sort 0 correlationID event
``` Find where the grouping changes ```
| streamstats count by grouping reset_on_change=t global=f
``` Assign the events to sequence groups ```
| streamstats count(eval(count == 1)) as sequence
``` Gather the field values by sequence group ```
| stats first(Date) as start last(Date) as end list(message) as message by sequence action_https correlationID loglevel
``` Reset Date field to first Date ```
| eval Date=start
``` Calculate the duration from the start and end times ```
| eval duration=round(1000*(strptime(end,"[%F %T,%3N]")-strptime(start,"[%F %T,%3N]")),0)
``` Sort results by Date (this a string based sort and works because of the date format used) ```
| sort 0 Date
``` Output table of fields ```
| table Date, loglevel, action_https, correlationID, message, duration

View solution in original post

0 Karma
Reply
Solved! Jump to solution

ITWhisperer
SplunkTrust
SplunkTrust
‎03-26-2024 05:00 AM

First of all, thank you for a good(-ish) description of your issue.

Try something like this:

| rex "(?<Date>\[[^\]]+\])\s(?<loglevel>\w+)\s-\swire\s(?<action_https>\S+)\sI\/O\s(?<correlationID>\S+)\s(?<direction>\S+)\s(?<message>.*)"
| eval grouping=correlationID.direction
| stats first(Date) as start last(Date) as end list(message) as message by grouping action_https correlationID loglevel
| eval Date=start
| eval duration=round(1000*(strptime(end,"[%F %T,%3N]")-strptime(start,"[%F %T,%3N]")),0)
| sort 0 Date
| table Date, loglevel, action_https, correlationID, message, duration

Note that your example shows unique combinations of correlationIDs and direction. If these are reused in your actual log, you may not get the results you expect. If so, please share a more representative version of your logs.

Reply
Solved! Jump to solution

michaelteck
Explorer
‎03-26-2024 06:51 AM

Hello @ITWhisperer, 

 

First of all, I'd like to thank you for taking the time to think about my concerns.
As you said, If the  combinations of correlationIDs and direction are reused it may not give the results I expect.
The correlationID and direction are completely random.
The correlationID is an ID that SWO2-APIM associates with the request to identify it.
The direction means that SWO2-APIM receives or sends the request.
In the real log, the first log line is at the bottom and the last log line is at the top.

This is the real logs look like :

[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "[\r][\n]"
[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "0[\r][\n]"
[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "<Message or something>[\r][\n]"
[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "8e[\r][\n]"
[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "[\r][\n]"
[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "Connection: close[\r][\n]"
[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "Transfer-Encoding: chunked[\r][\n]"
[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "Date: Tue, 26 Mar 2024 13:02:16 GMT[\r][\n]"
[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "Content-Type: application/xml; charset=UTF-8[\r][\n]"
[2024-03-26 13:02:16,357] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "HTTP/1.1 200 OK[\r][\n]"
[2024-03-26 13:02:16,353] DEBUG - wire HTTPS-Listener I/O dispatcher-4 >> "[\r][\n]"
[2024-03-26 13:02:16,353] DEBUG - wire HTTPS-Listener I/O dispatcher-4 >> "Accept-Encoding: gzip, compressed[\r][\n]"
[2024-03-26 13:02:16,353] DEBUG - wire HTTPS-Listener I/O dispatcher-4 >> "User-Agent: HealthChecker/2.0[\r][\n]"
[2024-03-26 13:02:16,353] DEBUG - wire HTTPS-Listener I/O dispatcher-4 >> "Connection: close[\r][\n]"
[2024-03-26 13:02:16,353] DEBUG - wire HTTPS-Listener I/O dispatcher-4 >> "Host: 10.229.55.71:8243[\r][\n]"
[2024-03-26 13:02:16,353] DEBUG - wire HTTPS-Listener I/O dispatcher-4 >> "GET /services/Version HTTP/1.1[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "0[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "<Message or something>[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "8e[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "Connection: close[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "Transfer-Encoding: chunked[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "Date: Tue, 26 Mar 2024 13:02:11 GMT[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "Content-Type: application/xml; charset=UTF-8[\r][\n]"
[2024-03-26 13:02:11,042] DEBUG - wire HTTPS-Listener I/O dispatcher-3 << "HTTP/1.1 200 OK[\r][\n]"
[2024-03-26 13:02:07,131] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "[\r][\n]"
[2024-03-26 13:02:07,131] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "0[\r][\n]"
[2024-03-26 13:02:07,131] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "<Message or something>[\r][\n]"
[2024-03-26 13:02:07,131] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "8e[\r][\n]"
[2024-03-26 13:02:07,131] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "[\r][\n]"
[2024-03-26 13:02:07,131] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "Connection: close[\r][\n]"	
[2024-03-26 13:02:07,131] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "Transfer-Encoding: chunked[\r][\n]"	
[2024-03-26 13:02:07,129] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "Date: Tue, 26 Mar 2024 13:02:07 GMT[\r][\n]"	
[2024-03-26 13:02:07,129] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "Content-Type: application/xml; charset=UTF-8[\r][\n]"	
[2024-03-26 13:02:07,129] DEBUG - wire HTTPS-Listener I/O dispatcher-4 << "HTTP/1.1 200 OK[\r][\n]"

 

If you look closely at the requests, they are received from bottom to top.
And so, I would like to have this kind of outing :

Date, loglevel, action_https, correlationID, message, duration
[2024-03-26 13:02:16,357], DEBUG, HTTPS-Listener, dispatcher-4, "HTTP/1.1 200 OK[\r][\n]" "Content-Type: application/xml; charset=UTF-8[\r][\n]" "Date: Tue, 26 Mar 2024 13:02:16 GMT[\r][\n]" "Transfer-Encoding: chunked[\r][\n]" "Connection: close[\r][\n]" "[\r][\n]" "8e[\r][\n]" "<Message or something>[\r][\n]" "0[\r][\n]" "[\r][\n]", 000
[2024-03-26 13:02:16,353], DEBUG, HTTPS-Listener, dispatcher-4, "GET /services/Version HTTP/1.1[\r][\n]" "Host: 10.229.55.71:8243[\r][\n]" "Connection: close[\r][\n]" "User-Agent: ELB-HealthChecker/2.0[\r][\n]" "Accept-Encoding: gzip, compressed[\r][\n]" "[\r][\n]", 000
[2024-03-26 13:02:11,042], DEBUG, HTTPS-Listener, dispatcher-3, "HTTP/1.1 200 OK[\r][\n]" "Content-Type: application/xml; charset=UTF-8[\r][\n]" "Date: Tue, 26 Mar 2024 13:02:11 GMT[\r][\n]" "Transfer-Encoding: chunked[\r][\n]" "Connection: close[\r][\n]" "[\r][\n]" "8e[\r][\n]" "<Message or something>[\r][\n]" "0[\r][\n]" "[\r][\n]", 000
[2024-03-26 13:02:07,129], DEBUG, HTTPS-Listener, dispatcher-4, "HTTP/1.1 200 OK[\r][\n]" "Content-Type: application/xml; charset=UTF-8[\r][\n]" "Date: Tue, 26 Mar 2024 13:02:07 GMT[\r][\n]" "Transfer-Encoding: chunked[\r][\n]" "Connection: close[\r][\n]" "[\r][\n]" "8e[\r][\n]" "<Message or something>[\r][\n]" "0[\r][\n]" "[\r][\n]", 003
0 Karma
Reply
Solved! Jump to solution
Solution

ITWhisperer
SplunkTrust
SplunkTrust
‎03-26-2024 07:56 AM

Try something like this

``` Reverse the order of events so that earliest is first ```
| reverse
``` Extract the fields ```
| rex "(?<Date>\[[^\]]+\])\s(?<loglevel>\w+)\s-\swire\s(?<action_https>\S+)\sI\/O\s(?<correlationID>\S+)\s(?<direction>\S+)\s(?<message>.*)"
``` Tag the events in order to be able to maintain the sequence ```
| streamstats count as event
``` Create a direction-based grouping for correlatoinIds ```
| eval grouping=correlationID.direction
``` Sort so that events for the same correlationID are together and in sequence ```
| sort 0 correlationID event
``` Find where the grouping changes ```
| streamstats count by grouping reset_on_change=t global=f
``` Assign the events to sequence groups ```
| streamstats count(eval(count == 1)) as sequence
``` Gather the field values by sequence group ```
| stats first(Date) as start last(Date) as end list(message) as message by sequence action_https correlationID loglevel
``` Reset Date field to first Date ```
| eval Date=start
``` Calculate the duration from the start and end times ```
| eval duration=round(1000*(strptime(end,"[%F %T,%3N]")-strptime(start,"[%F %T,%3N]")),0)
``` Sort results by Date (this a string based sort and works because of the date format used) ```
| sort 0 Date
``` Output table of fields ```
| table Date, loglevel, action_https, correlationID, message, duration
0 Karma
Reply
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...