How can I combine several transactions into one search and chart

[kaurinko]

Hi,

I am trying to get some performance/profiling statistics from our system. The log is very elar and aesy to read with something like this:

index=myindex "GetData : Request received." OR "GetData : Sending response."
| transaction TRXID maxspan=5m startswith="GetData : Request received." endswith="GetData : Sending response."
| timechart avg(duration) AS "GetData average"

What I would like to get is a similar one with somewhat different keywords, and illustrate the data simultaneously. As is obvious, these transactions are parts of a bigger round trip and I would like to profile, where the time is spent. The individual searches are in effect similar, but how can I use them simultaneously?

[kaurinko · Accepted solution]

I solved it, at least to some extent.

index=myindex ("GetData : Request received." OR "GetData : Sending response.")
| transaction TRXID maxspan=5m endswith="GetData : Sending response." startswith="GetData : Request received."
| timechart perc95(duration) AS "Data"
| append [
search index=myindex ("CheckServices : Request received." OR "CheckServices : Sending response.")
| transaction TRXID maxspan=5m endswith="CheckServices : Sending response." startswith="CheckServices : Request received." 
| timechart perc95(duration) AS "Srv" 
] 
| append [
search myindex ("CheckPermissions : Request received." OR "CheckPermissions : Sending response.")
| transaction TRXID maxspan=5m endswith="CheckPermissions : Sending response." startswith="CheckPermissions : Request received." 
| timechart perc95(duration) AS "Perm" 
] | timechart avg(Data) AS "95% Data" avg(Srv) AS "95% Services" avg(Perm) AS "95% Permissions"

So the solution is to search though the data separately for each transaction type, and my fear is that this is expensive for Splunk. Also, the data I get to the plot is what I request in the inner timechart requests. The last timechart does not change the 95%-data to averages, maybe because it has already been processed and reduced to one datapoint for each time span instead of the actual dataset. The last line gives the serieses their names, and without the last timechart, there would be three plots next to each other.

Any suggestions on how to improve the search in terms of performance without loosing clarity are gratefully appreciated.

[aaraneta_splunk]

Hi @kaurinko - Glad to see that you've found a potential solution yourself. If you'd like to close out this question, please click "Accept" below your answer. But if you would still like to keep it open in case another user wants to attempt to provide an alternative solution, you don't need to do anything. Thanks.

[sundareshr]

With the transaction command, I'm not sure you can combine multple sources without using sub-search (append). Instead, I would suggest not using sub-searches and do something like this

index=myindex ("GetData : Request received." OR "GetData : Sending response." OR "Someother critearia: Start" OR "Someother critearia: End") | stats earliest(eval(if(match(_raw, "GetData\s:\sRequest\sreceived"), _time, null()))) as req_recd earliest(eval(if(match(_raw, "GetData\s:\sSending\sresponse"), _time, null()))) as send_resp earliest(eval(if(match(_raw, "Someother critearia: Start"), _time, null()))) as start earliest(eval(if(match(_raw, "Someother critearia: End"), _time, null()))) as start by TRXID | eval duration_getdata= send_resp - req_recd | eval dur=end-start | timechart avg(duration_getdata) as avg_getdata avg(dur) as someothercriteria

[kaurinko]

I tried this, but I could not get anything to a timechart. The statistics seemed to come out by TRXID, but my Splunk-skills were insufficient to transform that to a graph. Leaving the last timechart out gave me the numerics, but I was after the graphical time-history.