We are using Splunk to monitor the traffic of our system, and i was asked to give a report for showing the moving maximum concurrency count in second level of the recent 30 days, for example:
**_time=5/9/10 12:00:01,Type=UI, ReqTime=100 _time=5/9/10 12:00:01,Type=UI, ReqTime=1 _time=5/9/10 12:00:01,Type=UI, ReqTime=30 _time=5/9/10 12:00:01,Type=UI, ReqTime=9** _time=5/9/10 12:00:02,Type=UI, ReqTime=5
There are 4 concurrency requests at "5/9/10 12:00:01"
_time=5/10/10 1:00:01,Type=UI, ReqTime=3 _time=5/10/10 1:00:01,Type=UI, ReqTime=2 _time=5/10/10 1:00:01,Type=UI, ReqTime=50 _time=5/10/10 12:00:01,Type=UI, ReqTime=0 _time=5/10/10 12:00:02,Type=UI, ReqTime=1
There are 3 concurrency requests at "5/10/10 1:00:01"
... day -30
I want to have chart like this:
Day Max concurrency count per second Time 5/9/2010 4 5/9/10 12:00:01 5/10/2010 3 5/10/10 1:00:01 ...
thanks in advance!
This is a very interesting question and none of the answers actually get you to the full solution, where you find not only the top concurrency during the day, but when that occurred.
First you should note that the
bin command simply flattens one field into "bins". For example, it turns seconds of the day into hours of the day. It alone cannot do the job here,it must be combined with
concurrency to find the actual concurrency at any transition point in the time series (i.e., when a connection comes or goes).
Here's the first search, which will find the most recent occurrence of the top concurrency time of the day:
... | eval duration = ReqTime/1000 | concurrency duration=duration | bin span=dh _time as day | dedup day sortby -concurrency | eval when=_time | timechart span=1d max(concurrency) as concurrency list(when) as when | convert ctime(when)
Now your actual search is a bit trickier since we may have many points in the day of highest concurrency:
... | eval duration = ReqTime/1000 | concurrency duration=duration | bin span=1d _time as day | eventstats max(concurrency) as max_concurrency by day | eval when=if(concurrency==max_concurrency, _time, null()) | timechart span=1d max(concurrency) as concurrency list(when) as when | convert ctime(when)
The general recipe here is:
dedupto find the single highest concurrency point, or
eventstats+evalto find the many highest equal concurrency points.
Note that this technique can be used on splunk_access.log data using the following search:
index=_internal source=*/splunkd_access.log earliest=-24h@h | eval duration = spent/1000 | concurrency duration=duration | bin span=1h _time as hour | eventstats max(concurrency) as max_concurrency by hour | eval when=if(concurrency==max_concurrency, _time, null()) | timechart span=1h max(concurrency) as concurrency list(when) as when | convert ctime(when)
I just counted the requests by using grep from the log file directly, seems you are right the first expression wasn't give the right answer here, so i have two questions for you:
a).why the first one is wrong, i'm stupid and i use it was because i saw an example from the document
"Return the average "thruput" of each "host" for each 5 minute time span.
... | bucket _time span=5m | stats avg(thruput) by _time host"
so i think use it can put all events into discrete sets by using 1sec as time range, and then doing count for each set should get the right answer. ( i knew i didn't consider the ReqTime for calculating concurrency, however in my opinion, the result should match with the "grep")
b). If the second one is right, how can i set the range to 1 sec instead of 10 minutes
_time max(concurrency) 1 5/16/10 5:**50**:00.000 PM 111 2 5/16/10 5:**40**:00.000 PM 141
and what's the unit of "duration"? the unit of ReqTime is msec in our system.
if ReqTime is ms and not seconds, just do
| eval ReqTimeSecs=ReqTime/1000 | concurrency duration=ReqTimeSecs. Duration, like
duration and most other times in Splunk, is in seconds.
I don't understand what the "range" you are looking for is. If it's just the highest value of "concurrency" in a given span, then use the "span=1s" parameter of "timechart".
Thanks for your quick responses!
I just got a chance to try "concurrency" in my local, however the result wasn't make sense to me, the case i was using for testing is
"Showing the max concurrency count in second level of day '05/16/2010'"
and i tried to use two expression to figure it out:
index=main Type=API earliest=05/16/2010:6:0:0 latest=05/16/2010:18:0:0 | bucket _time span=1s | stats count(_raw) by _time | sort -count(_raw) | head 1
_time count(_raw) 1 5/16/10 5:35:36.000 PM 28
index=main Type=API earliest=05/16/2010:6:0:0 latest=05/16/2010:18:0:0 | concurrency duration=ReqTime | timechart max(concurrency)
max(concurrency) 1 5/16/10 6:00:00.000 AM 47
the result was different, i'm not sure which one is right, do you think the expression i used is make sense?
Indeed this is exactly the sort of use case that the concurrency command was added for:
<your search here> | concurrency duration=ReqTime | timechart max(concurrency)
Concurrency is only in 4.1. If you havent upgraded to 4.1 yet, it's possible there was a difficult and somewhat evil way of achieving the same thing with other search language and without
concurrency but i dont know what it was.