Splunk Search

How can I correlate a variable number of fields with a similar prefix in a single event?

kalldrexx
Observer

We have a web api that orchestrates calls to other services. So for example we may have an incoming call to `/api`, which then may do 3 calls to a system of record. These executions are ending up in splunk in a single metric (mostly for ease of investigation in other avenues). So we have events that look like the following:

 

 

{
    SOR.Executions.0.Operation: "GetInfo",
    SOR.Executions.0.TimeInMs: 321,
    SOR.Executions.1.Operation: "UpdateRecord",
    SOR.Exectuions.1.TimeInMs: 234,
    SOR.Executions.2.Operation: "DoSomethingElse",
    SOR.Exectuions.2.TimeInMs: 532,
}

 

 

 

I've been able to successfully extract singular values from these via a search such as

 

 

index="docker" 
| fields SOR.Executions.*.Operation
| foreach SOR.Executions.*.Operation [eval Operation=mvappend(VcmRequest, '<<FIELD>>')]
| mvexpand Operation
| fields Operation
| stats count by Operation
| sort count desc

 

 

 

This works well for retrieving singular values without correlations (and allows for nice pie charts for how often specific operations are performed) but now I want to get percentile timings of each individual operation.  So for example I want to run statistics on how often a "GetInfo" operation takes vs how often a "DoSomethingEsle" operation takes. 

 

The problem is that the number of SOR calls being made varies depending on input to the API call.  We may end up with 3 SOR calls if 1 customer is passed in, or we may end up with 8 calls if 5 customer ids are passed in. 

 

My initial thought was to do an eval to grab the field name and put it in a multi-value field, then do a `rex` to pull out the execution digit (the dynamic part), tokenizing that, then doing a subsearch based on that.  However, that's getting to things I can't find documentation of (pulling out field names for example) and is going into a complex rabbit hole.

 

Is there an easier avenue to do this that I'm missing?

Labels (1)
0 Karma

kalldrexx
Observer

Figured it out!  The trick I was missing was the `mvzip` function

 

 

index="docker" sourcetype="openshift" kubernetes.container_name="kirbyapi*" MDM.ExecutionCount>0
| fields SOR.Executions.*.Data.SORRequestType.*  SOR.Executions.*.TimeInMs
| foreach SOR.Executions.*.Data.SORRequestType.* [eval SORRequest=mvappend(SORRequest, '<<FIELD>>')]
| foreach SOR.Executions.*.TimeInMs [eval SORTimeInMs=mvappend(SORTimeInMs, '<<FIELD>>')]
| eval SORRequestWithTimes=mvzip(SORRequest, SORTimeInMs)
| mvexpand SORRequestWithTimes
| rex field=SORRequestWithTimes "(?<SORRequestType>^.*),(?<SORRequestTime>.*)"
| fields SORRequestType SORRequestTime
| stats count as Requests 
perc50(SORRequestTime) as P50 
perc90(SORRequestTime) as P90  
perc95(SORRequestTime) as P95
perc99(SORRequestTime) as P99
by SORRequestType
| sort Requests desc

 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...