topic How to get P90 latency from combine trace_ids of one query to be fed to a second query. in Splunk Search

How to get P90 latency from combine trace_ids of one query to be fed to a second query.

cimino — Fri, 06 Sep 2024 23:07:35 GMT

If I have two queries:

index=poc container_name=app horizontalId=orange

outputs events with the trace ids

index=poc container_name=app ExecutionTimeAspect Elastic Vertical Search Query Service | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

outputs a Latency = 845

I want to link output of query 2 and query 1 via the trace ids for the P90 Latency.

Re: How to get P90 latency from combine trace_ids of one query to be fed to a second query.

ITWhisperer — Sat, 07 Sep 2024 08:51:32 GMT

What does "link" mean in this context? The second query doesn't return any trace ids. Please clarify what you are trying to do (in non=SPL terms, provide some sample events, and a representation of your expected output.

Re: How to get P90 latency from combine trace_ids of one query to be fed to a second query.

cimino — Sat, 07 Sep 2024 16:04:03 GMT

Thanks for quick response.

Link means to combine trace_ids of the first query and fed into the second query. Ex. take the trace ids output from the first query and add it to the second query for the P90 search latency total. The first query returns trace_ids

outputs look like this

2024-... 15:23:58.961 INFO c.....impl....r#58 - Response from a....: ... [service.name=<service-name>=qa,trace_id=2b......,span_id=cs.....,trace_flags=01]

P90 Latency query

index=<> container-name=<> Exec... Search Query Service | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

if I want to combine the output of query 1 via trace ids, how can I do that so that the query 2 is the latency value?

Re: How to get P90 latency from combine trace_ids of one query to be fed to a second query.

ITWhisperer — Sat, 07 Sep 2024 20:40:47 GMT

Do you mean something like this?

index=poc container_name=app ExecutionTimeAspect Elastic Vertical Search Query Service [ search index=poc container_name=app horizontalId=orange | stats count by trace_id | table trace_id] | rex field=_raw "execution time is[ ]+(?<latency>\d+)[ ]+ms" | stats p90(latency) as Latency

Re: How to get P90 latency from combine trace_ids of one query to be fed to a second query.

cimino — Tue, 10 Sep 2024 13:48:26 GMT

The output is numerical with the inner search query. To validate this output, the next step is to check the p90 latencies in Splunk Observability Cloud for these traces and compare the values. Thank you.

Re: How to get P90 latency from combine trace_ids of one query to be fed to a second query.

ITWhisperer — Tue, 10 Sep 2024 13:51:36 GMT

Please provide some anonymised sample events for both searches and what your expected output would look like