Splunk Search

How to get P90 latency from combine trace_ids of one query to be fed to a second query.

cimino
Engager

If I have two queries:

1.

index=poc container_name=app horizontalId=orange

outputs events with the trace ids

2.

index=poc container_name=app ExecutionTimeAspect Elastic Vertical Search Query Service | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

outputs a Latency = 845

I want to link output of query 2 and query 1 via the trace ids for the P90 Latency.

Labels (1)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

Please provide some anonymised sample events for both searches and what your expected output would look like

View solution in original post

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What does "link" mean in this context? The second query doesn't return any trace ids. Please clarify what you are trying to do (in non=SPL terms, provide some sample events, and a representation of your expected output.

0 Karma

cimino
Engager

Thanks for quick response.

Link means to combine trace_ids of the first query and fed into the second query. Ex. take the trace ids output from the first query and add it to the second query for the P90 search latency total.  The first query returns trace_ids 

 

outputs look like this 

2024-... 15:23:58.961 INFO c.....impl....r#58 - Response from a....: ... [service.name=<service-name>=qa,trace_id=2b......,span_id=cs.....,trace_flags=01]

P90 Latency query

index=<> container-name=<> Exec... Search Query Service | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

if I want to combine the output of query 1 via trace ids, how can I do that so that the query 2 is the latency value?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Do you mean something like this?

index=poc container_name=app ExecutionTimeAspect Elastic Vertical Search Query Service [ search index=poc container_name=app horizontalId=orange | stats count by trace_id | table trace_id] | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency
0 Karma

cimino
Engager

The output is numerical with the inner search query. To validate this output, the next step is to check the p90 latencies in Splunk Observability Cloud for these traces and compare the values. Thank you.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Please provide some anonymised sample events for both searches and what your expected output would look like

0 Karma
Get Updates on the Splunk Community!

.conf25 Community Recap

Hello Splunkers, And just like that, .conf25 is in the books! What an incredible few days — full of learning, ...

Splunk App Developers | .conf25 Recap & What’s Next

If you stopped by the Builder Bar at .conf25 this year, thank you! The retro tech beer garden vibes were ...

Congratulations to the 2025-2026 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...