Splunk Search

How can I correlate a variable number of fields with a similar prefix in a single event?

kalldrexx
Observer

We have a web api that orchestrates calls to other services. So for example we may have an incoming call to `/api`, which then may do 3 calls to a system of record. These executions are ending up in splunk in a single metric (mostly for ease of investigation in other avenues). So we have events that look like the following:

 

 

{
    SOR.Executions.0.Operation: "GetInfo",
    SOR.Executions.0.TimeInMs: 321,
    SOR.Executions.1.Operation: "UpdateRecord",
    SOR.Exectuions.1.TimeInMs: 234,
    SOR.Executions.2.Operation: "DoSomethingElse",
    SOR.Exectuions.2.TimeInMs: 532,
}

 

 

 

I've been able to successfully extract singular values from these via a search such as

 

 

index="docker" 
| fields SOR.Executions.*.Operation
| foreach SOR.Executions.*.Operation [eval Operation=mvappend(VcmRequest, '<<FIELD>>')]
| mvexpand Operation
| fields Operation
| stats count by Operation
| sort count desc

 

 

 

This works well for retrieving singular values without correlations (and allows for nice pie charts for how often specific operations are performed) but now I want to get percentile timings of each individual operation.  So for example I want to run statistics on how often a "GetInfo" operation takes vs how often a "DoSomethingEsle" operation takes. 

 

The problem is that the number of SOR calls being made varies depending on input to the API call.  We may end up with 3 SOR calls if 1 customer is passed in, or we may end up with 8 calls if 5 customer ids are passed in. 

 

My initial thought was to do an eval to grab the field name and put it in a multi-value field, then do a `rex` to pull out the execution digit (the dynamic part), tokenizing that, then doing a subsearch based on that.  However, that's getting to things I can't find documentation of (pulling out field names for example) and is going into a complex rabbit hole.

 

Is there an easier avenue to do this that I'm missing?

Labels (1)
0 Karma

kalldrexx
Observer

Figured it out!  The trick I was missing was the `mvzip` function

 

 

index="docker" sourcetype="openshift" kubernetes.container_name="kirbyapi*" MDM.ExecutionCount>0
| fields SOR.Executions.*.Data.SORRequestType.*  SOR.Executions.*.TimeInMs
| foreach SOR.Executions.*.Data.SORRequestType.* [eval SORRequest=mvappend(SORRequest, '<<FIELD>>')]
| foreach SOR.Executions.*.TimeInMs [eval SORTimeInMs=mvappend(SORTimeInMs, '<<FIELD>>')]
| eval SORRequestWithTimes=mvzip(SORRequest, SORTimeInMs)
| mvexpand SORRequestWithTimes
| rex field=SORRequestWithTimes "(?<SORRequestType>^.*),(?<SORRequestTime>.*)"
| fields SORRequestType SORRequestTime
| stats count as Requests 
perc50(SORRequestTime) as P50 
perc90(SORRequestTime) as P90  
perc95(SORRequestTime) as P95
perc99(SORRequestTime) as P99
by SORRequestType
| sort Requests desc

 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Community Content Calendar, November Edition

Welcome to the November edition of our Community Spotlight! Each month, we dive into the Splunk Community to ...

October Community Champions: A Shoutout to Our Contributors!

As October comes to a close, we want to take a moment to celebrate the people who make the Splunk Community ...

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

What are Community Office Hours? Community Office Hours is an interactive 60-minute Zoom series where ...