Splunk Search

How to get P90 latency from combine trace_ids of one query to be fed to a second query.

cimino
Engager

If I have two queries:

1.

index=poc container_name=app horizontalId=orange

outputs events with the trace ids

2.

index=poc container_name=app ExecutionTimeAspect Elastic Vertical Search Query Service | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

outputs a Latency = 845

I want to link output of query 2 and query 1 via the trace ids for the P90 Latency.

Labels (1)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

Please provide some anonymised sample events for both searches and what your expected output would look like

View solution in original post

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What does "link" mean in this context? The second query doesn't return any trace ids. Please clarify what you are trying to do (in non=SPL terms, provide some sample events, and a representation of your expected output.

0 Karma

cimino
Engager

Thanks for quick response.

Link means to combine trace_ids of the first query and fed into the second query. Ex. take the trace ids output from the first query and add it to the second query for the P90 search latency total.  The first query returns trace_ids 

 

outputs look like this 

2024-... 15:23:58.961 INFO c.....impl....r#58 - Response from a....: ... [service.name=<service-name>=qa,trace_id=2b......,span_id=cs.....,trace_flags=01]

P90 Latency query

index=<> container-name=<> Exec... Search Query Service | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

if I want to combine the output of query 1 via trace ids, how can I do that so that the query 2 is the latency value?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Do you mean something like this?

index=poc container_name=app ExecutionTimeAspect Elastic Vertical Search Query Service [ search index=poc container_name=app horizontalId=orange | stats count by trace_id | table trace_id] | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency
0 Karma

cimino
Engager

The output is numerical with the inner search query. To validate this output, the next step is to check the p90 latencies in Splunk Observability Cloud for these traces and compare the values. Thank you.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Please provide some anonymised sample events for both searches and what your expected output would look like

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

.conf25 Global Broadcast: Don’t Miss a Moment

Hello Splunkers, .conf25 is only a click away.  Not able to make it to .conf25 in person? No worries, you can ...

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...