Splunk Search

How can I combine several transactions into one search and chart

kaurinko
Communicator

Hi,

I am trying to get some performance/profiling statistics from our system. The log is very elar and aesy to read with something like this:

index=myindex "GetData : Request received." OR "GetData : Sending response."
| transaction TRXID maxspan=5m startswith="GetData : Request received." endswith="GetData : Sending response."
| timechart avg(duration) AS "GetData average"

What I would like to get is a similar one with somewhat different keywords, and illustrate the data simultaneously. As is obvious, these transactions are parts of a bigger round trip and I would like to profile, where the time is spent. The individual searches are in effect similar, but how can I use them simultaneously?

0 Karma
1 Solution

kaurinko
Communicator

I solved it, at least to some extent.

index=myindex ("GetData : Request received." OR "GetData : Sending response.")
| transaction TRXID maxspan=5m endswith="GetData : Sending response." startswith="GetData : Request received."
| timechart perc95(duration) AS "Data"
| append [
search index=myindex ("CheckServices : Request received." OR "CheckServices : Sending response.")
| transaction TRXID maxspan=5m endswith="CheckServices : Sending response." startswith="CheckServices : Request received." 
| timechart perc95(duration) AS "Srv" 
] 
| append [
search myindex ("CheckPermissions : Request received." OR "CheckPermissions : Sending response.")
| transaction TRXID maxspan=5m endswith="CheckPermissions : Sending response." startswith="CheckPermissions : Request received." 
| timechart perc95(duration) AS "Perm" 
] | timechart avg(Data) AS "95% Data" avg(Srv) AS "95% Services" avg(Perm) AS "95% Permissions"

So the solution is to search though the data separately for each transaction type, and my fear is that this is expensive for Splunk. Also, the data I get to the plot is what I request in the inner timechart requests. The last timechart does not change the 95%-data to averages, maybe because it has already been processed and reduced to one datapoint for each time span instead of the actual dataset. The last line gives the serieses their names, and without the last timechart, there would be three plots next to each other.

Any suggestions on how to improve the search in terms of performance without loosing clarity are gratefully appreciated.

View solution in original post

0 Karma

kaurinko
Communicator

I solved it, at least to some extent.

index=myindex ("GetData : Request received." OR "GetData : Sending response.")
| transaction TRXID maxspan=5m endswith="GetData : Sending response." startswith="GetData : Request received."
| timechart perc95(duration) AS "Data"
| append [
search index=myindex ("CheckServices : Request received." OR "CheckServices : Sending response.")
| transaction TRXID maxspan=5m endswith="CheckServices : Sending response." startswith="CheckServices : Request received." 
| timechart perc95(duration) AS "Srv" 
] 
| append [
search myindex ("CheckPermissions : Request received." OR "CheckPermissions : Sending response.")
| transaction TRXID maxspan=5m endswith="CheckPermissions : Sending response." startswith="CheckPermissions : Request received." 
| timechart perc95(duration) AS "Perm" 
] | timechart avg(Data) AS "95% Data" avg(Srv) AS "95% Services" avg(Perm) AS "95% Permissions"

So the solution is to search though the data separately for each transaction type, and my fear is that this is expensive for Splunk. Also, the data I get to the plot is what I request in the inner timechart requests. The last timechart does not change the 95%-data to averages, maybe because it has already been processed and reduced to one datapoint for each time span instead of the actual dataset. The last line gives the serieses their names, and without the last timechart, there would be three plots next to each other.

Any suggestions on how to improve the search in terms of performance without loosing clarity are gratefully appreciated.

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

Hi @kaurinko - Glad to see that you've found a potential solution yourself. If you'd like to close out this question, please click "Accept" below your answer. But if you would still like to keep it open in case another user wants to attempt to provide an alternative solution, you don't need to do anything. Thanks.

sundareshr
Legend

With the transaction command, I'm not sure you can combine multple sources without using sub-search (append). Instead, I would suggest not using sub-searches and do something like this

index=myindex ("GetData : Request received." OR "GetData : Sending response." OR "Someother critearia: Start" OR "Someother critearia: End") | stats earliest(eval(if(match(_raw, "GetData\s:\sRequest\sreceived"), _time, null()))) as req_recd earliest(eval(if(match(_raw, "GetData\s:\sSending\sresponse"), _time, null()))) as send_resp earliest(eval(if(match(_raw, "Someother critearia: Start"), _time, null()))) as start earliest(eval(if(match(_raw, "Someother critearia: End"), _time, null()))) as start by TRXID | eval duration_getdata= send_resp - req_recd | eval dur=end-start | timechart avg(duration_getdata) as avg_getdata avg(dur) as someothercriteria
0 Karma

kaurinko
Communicator

I tried this, but I could not get anything to a timechart. The statistics seemed to come out by TRXID, but my Splunk-skills were insufficient to transform that to a graph. Leaving the last timechart out gave me the numerics, but I was after the graphical time-history.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...