How to get P90 latency from combine trace_ids of one query to be fed to a second query.

[cimino]

If I have two queries:

1.

index=poc container_name=app horizontalId=orange

outputs events with the trace ids

2.

index=poc container_name=app ExecutionTimeAspect Elastic Vertical Search Query Service | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

outputs a Latency = 845

I want to link output of query 2 and query 1 via the trace ids for the P90 Latency.

[ITWhisperer]

What does "link" mean in this context? The second query doesn't return any trace ids. Please clarify what you are trying to do (in non=SPL terms, provide some sample events, and a representation of your expected output.

[cimino]

Thanks for quick response.

Link means to combine trace_ids of the first query and fed into the second query. Ex. take the trace ids output from the first query and add it to the second query for the P90 search latency total.  The first query returns trace_ids 

outputs look like this 

2024-... 15:23:58.961 INFO c.....impl....r#58 - Response from a....: ... [service.name=<service-name>=qa,trace_id=2b......,span_id=cs.....,trace_flags=01]

P90 Latency query

index=<> container-name=<> Exec... Search Query Service | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

if I want to combine the output of query 1 via trace ids, how can I do that so that the query 2 is the latency value?

[ITWhisperer]

Do you mean something like this?

index=poc container_name=app ExecutionTimeAspect Elastic Vertical Search Query Service [ search index=poc container_name=app horizontalId=orange | stats count by trace_id | table trace_id] | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

[cimino]

The output is numerical with the inner search query. To validate this output, the next step is to check the p90 latencies in Splunk Observability Cloud for these traces and compare the values. Thank you.

[ITWhisperer · Accepted solution]

Please provide some anonymised sample events for both searches and what your expected output would look like