Splunk Concurrency Calculation with Drilldown

SBB — Tue, 05 Oct 2021 04:38:35 GMT

I have some data from logs in Splunk where I need to determine what other requests were running concurrently at the time of any single event.

Using the following query, I was able to have it return a column for the number of requests that ran at the same time within my start time and duration.

index="sfdc" source="sfdc_event_log://EventLog_SFDC_Production_eventlog_hourly" EVENT_TYPE IN (API, RestAPI) RUN_TIME>20000 
| eval endTime=_time 
| eval permitTimeInSecs=(RUN_TIME-20000)/1000 
| eval permitAcquiredTime=endTime-permitTimeInSecs
| eval dbTotalTime=DB_TOTAL_TIME/1000000
| concurrency start=permitAcquiredTime duration=permitTimeInSecs 
| table _time API_TYPE EVENT_TYPE ENTITY_NAME apimethod concurrency permitAcquiredTime permitTimeInSecs RUN_TIME CPU_TIME dbtotalTime REQUEST_ID USER_ID
| fieldformat dbTotalTime=round(dbTotalTime,0)
| rename permitAcquiredTime as "Start Time", permitTimeInSecs as "Concurrency Duration", concurrency as "Concurrent Running Events", API_TYPE as "API Type", EVENT_TYPE as "Event Type", ENTITY_NAME as "Entity Name", apimethod as "API Method", RUN_TIME as "Run Time", CPU_TIME as "CPU Time", dbtotalTime as "DB Total Time", REQUEST_ID as "Request ID", USER_ID as "User ID"
| sort "Concurrent Running Events" desc

6JZwR.png

I am now trying to investigate a single event in these results. For example, the top event says that at the time it ran, there were 108 concurrent requests running in the 20 second window of time.

How can I identify those 108 events using this data?

I imagine it would be querying the events that had a specific time frame range, but I am not sure if I need to check something like _time + - 10 seconds to see what was running within the 20 second window?

I just need to understand the data behind this 108 events a little more for this top example. My end goal here is to be able to add a drill-down to the dashboard so that when I click on the 108, I can see those events that were running concurrently.

topic Splunk Concurrency Calculation with Drilldown in Splunk Search

Splunk Concurrency Calculation with Drilldown