Splunk Search

Charting events in a transaction that delineates them

Explorer

I have events with numbers that I would like to chart, but only those that lie between a specific set of other events.

I can use a transaction to easily group the events I care about, but I can't figure out how to then chart those events' values.

Here's an example of the events:

server=server01 event=testcase.start testName=test01
server=server01 event=fps median=21.596733 finalFrameTime=130314411339538662
server=server01 event=fps median=21.466275 finalFrameTime=130314411340699548
server=server01 event=fps median=16.639793 finalFrameTime=130314411366595094
server=server01 event=testcase.end errors=0 testName=test01
server=server01 event=testcase.start testName=test02
server=server01 event=fps median=23.648486 finalFrameTime=130314411389632969
server=server01 event=fps median=20.461889 finalFrameTime=130314411391554905
server=server01 event=fps median=17.129764 finalFrameTime=130314411416874998
server=server01 event=testcase.end errors=0 testName=test02
server=server02 event=testcase.start testName=test02
server=server02 event=fps median=19.813126 finalFrameTime=130314411439709787
server=server02 event=fps median=25.649891 finalFrameTime=130314411441514007
server=server02 event=fps median=22.911548 finalFrameTime=130314411466902957
server=server02 event=testcase.end errors=0 testName=test02

I want to find the event=fps rows that are bookmarked by testName=test02 and then chart the "median" field over time.

I can easily use transactions to get the first part:

(event=testcase.* testName=test02) OR event=fps | transaction server startswith=event=testcase.start endswith=event=testcase.end maxSpan=45m

but then I want to chart the 6 values of "median" in the test02 fps events. My intent is to append something like this to the above:

... | eval _time=max(finalFrameTime)/10000000 - 11644473600 | timechart median(median)

but there are no fields named "finalFrameTime" or "median" any longer.

It seems like I need to now "break up" the transactions once their job of filtering out rows belonging to other tests is done.

Tags (1)
0 Karma
1 Solution

Motivator

EDIT

So the underscore mislead me for a little bit, it doesn't appear to make it work. It appears that mvlist works ok with 1 value, but not with a list of values. I tried both commas and spaces as separators for the mvlist values and neither worked. I think a workaround for now would be to use mvlist=t and only work with the fields you want.

END EDIT

Yes it does. This may be a stretch but... Looking back at your search command, I noticed that your rex which defines the new fields is all lower case but your eval and timechart commands have some uppercase characters. Field names are case sensitive in Splunk. If the search you are running has the same differences in case, can you try your search again with correct case in your rex statement?

Also, what version of Splunk are you using? I'm on version 6, and we may have found a bug here. I've tried running similar searches with similar results. It looks like the mvlist function is having some issues. Try running just this part of the search:

(event=testcase.* testName=test02) OR event=fps 
| transaction server startswith=event=testcase.start endswith=event=testcase.end maxSpan=45m mvlist=median,finalFrameTime 

At this point you should have several values for the median field and the finalFrameTime field inside each transaction. But, using my own data and running a similar search, when I run this I get many values for what would be the median field but only ONE value for what would be the finalFrameTime. HOWEVER, if I add an underscore somewhere in the finalFrameTime field in my mvlist command, it works!?!? I tried putting an underscore in various places on that field name and it works... I don't know why, but that makes it work on my system. Try this:

(event=testcase.* testName=test02) OR event=fps 
| transaction server startswith=event=testcase.start endswith=event=testcase.end maxSpan=45m mvlist=median,finalF_rameTime 

or this:

(event=testcase.* testName=test02) OR event=fps 
| transaction server startswith=event=testcase.start endswith=event=testcase.end maxSpan=45m mvlist=median,finalFrameT_ime 

and see what you get. If it works, slowly add in the rest of the search and make sure each part works before proceeding.

View solution in original post

Explorer

A different approach that I tried that seems to work is to use join to find fields from the start event:

event=fps
| join usetime=t earlier=t max=1 server [search (event="testcase.start")]
| where testName="test02"
| eval _time=max(finalFrameTime)/10000000 - 11644473600
| timechart median(median)

0 Karma

Motivator

EDIT

So the underscore mislead me for a little bit, it doesn't appear to make it work. It appears that mvlist works ok with 1 value, but not with a list of values. I tried both commas and spaces as separators for the mvlist values and neither worked. I think a workaround for now would be to use mvlist=t and only work with the fields you want.

END EDIT

Yes it does. This may be a stretch but... Looking back at your search command, I noticed that your rex which defines the new fields is all lower case but your eval and timechart commands have some uppercase characters. Field names are case sensitive in Splunk. If the search you are running has the same differences in case, can you try your search again with correct case in your rex statement?

Also, what version of Splunk are you using? I'm on version 6, and we may have found a bug here. I've tried running similar searches with similar results. It looks like the mvlist function is having some issues. Try running just this part of the search:

(event=testcase.* testName=test02) OR event=fps 
| transaction server startswith=event=testcase.start endswith=event=testcase.end maxSpan=45m mvlist=median,finalFrameTime 

At this point you should have several values for the median field and the finalFrameTime field inside each transaction. But, using my own data and running a similar search, when I run this I get many values for what would be the median field but only ONE value for what would be the finalFrameTime. HOWEVER, if I add an underscore somewhere in the finalFrameTime field in my mvlist command, it works!?!? I tried putting an underscore in various places on that field name and it works... I don't know why, but that makes it work on my system. Try this:

(event=testcase.* testName=test02) OR event=fps 
| transaction server startswith=event=testcase.start endswith=event=testcase.end maxSpan=45m mvlist=median,finalF_rameTime 

or this:

(event=testcase.* testName=test02) OR event=fps 
| transaction server startswith=event=testcase.start endswith=event=testcase.end maxSpan=45m mvlist=median,finalFrameT_ime 

and see what you get. If it works, slowly add in the rest of the search and make sure each part works before proceeding.

View solution in original post

Explorer

Yes mvlist=t seems to work and give the correct values.

0 Karma

Explorer

I think it's the formatting of this board on comments that made my rex command have all lowercase field names, when I did paste in mixed case.

I am using 5.0.2.

The result of the transaction command is some transactions with either 0 or 1 fps events listed. Here's an example of 1 fps event listed, even though there are several fps events that should be in the transaction [this is in the "Events List" view]:

server=server02 event=testcase.start testName=test02
server=server02 event=fps median=8.333260 finalFrameTime=130317012355621023
server=server02 event=testcase.end testName=test02

0 Karma

Motivator

I can't post comments from my workplace for some reason so I have to post this in a new answer.

Hmm, I don't know why newMedian wouldn't be showing up unless you add in another field, that's a strange one.

As to only having one value per transaction, maybe try "... | timechart count by newMedian" instead of "timechart median(newMedian)". The medain() function will return only the middle value of the field for that chunk of time. So, if your span is 15 minutes, median() will return the middle value of the field for that entire 15 minute span. If your span is automatic, it will still only return one value per chunk of time. To get what you want, I think you will need to change your aggregation term in timechart. Try using "... | timechart count by newMedian" and use a stacked column chart for a visualization? If you need to see more than 10 medians, add a limit to your timechart command.

0 Karma

Explorer

My current data set has 3 transactions, each with about 5 fps events.

When I do 'count by newMedian', it labels the series by the 3 values of newMedian and the three non-zero y values are each "1".

This seems to indicate that it is already in one value per transaction by the time it gets to the timechart command.

0 Karma

Motivator

Stay with me, this will get a little convoluted. We'll need to use mvexpand to create individual events for each combination of median and finalFrameTime but it will take some doing to get there since mvexpand only works on one multivalue field. Try using mvlist inside your transaction command for the median and finalFrameTime fields. Mvlist will keep these fields in the proper order inside your transaction. Next use mvzip to combine each value of median with finalFrameTime (we'll separate them later). After that, use mvexpand on the combined field to create a new, individual event for each combination of median and finalFrameTime. Next, separate out the values of median and finalFrameTime, then eval _time for each of your newly created events, and use it to report on:

(event=testcase.* testName=test02) OR event=fps 
| transaction server startswith=event=testcase.start endswith=event=testcase.end maxSpan=45m mvlist=median,finalFrameTime 
| mvzip combined=(median,finalFrameTime)
| mvexpand combined
| rex field=combined "(?<newMedian>[^,]+),(?<newFinalFrameTime>.*)"
| eval _time=max(newFinalFrameTime)/10000000 - 11644473600 
| timechart median(newMedian)

Something along those lines. I can't test this out where I'm at (no Splunk at my house)so this is off the top of my head, but I think this should get what you close to what you need.

Explorer

This only shows one plot point per transaction, instead of a plot point for each fps event in the transaction.

My actual use case is five numbers instead of just median, so that's how I discovered that this leaves off the first number, for some reason.

0 Karma

Explorer

For some reason the first field ["median"] is left off doing that, but if I put another field that I don't care about in ["frames"], I can get median:

(event=testcase.* testName=test02) OR event=fps
| transaction server startswith=event=testcase.start endswith=event=testcase.end maxSpan=45m mvlist=frames,median,finalFrameTime
| eval combined=mvzip(frames,mvzip(median,finalFrameTime)
| mvexpand combined
| rex field=combined "(?[^,]+),(?[^,]+),(?.*)"
| eval _time=max(newFinalFrameTime)/10000000 - 11644473600
| timechart median(newMedian)

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!