how to show a moving average in the recent 30days about the top concurrency count per second in each day

[KangKangZhu]

Hello,

We are using Splunk to monitor the traffic of our system, and i was asked to give a report for showing the moving maximum concurrency count in second level of the recent 30 days, for example:

day-1

**_time=5/9/10 12:00:01,Type=UI, ReqTime=100
_time=5/9/10 12:00:01,Type=UI, ReqTime=1
_time=5/9/10 12:00:01,Type=UI, ReqTime=30
_time=5/9/10 12:00:01,Type=UI, ReqTime=9**
_time=5/9/10 12:00:02,Type=UI, ReqTime=5

There are 4 concurrency requests at "5/9/10 12:00:01"

day-2

_time=5/10/10 1:00:01,Type=UI, ReqTime=3
_time=5/10/10 1:00:01,Type=UI, ReqTime=2
_time=5/10/10 1:00:01,Type=UI, ReqTime=50
_time=5/10/10 12:00:01,Type=UI, ReqTime=0
_time=5/10/10 12:00:02,Type=UI, ReqTime=1

There are 3 concurrency requests at "5/10/10 1:00:01"

... day -30

I want to have chart like this:

Day         Max concurrency count per second          Time
5/9/2010        4                               5/9/10 12:00:01
5/10/2010       3                               5/10/10 1:00:01
...

thanks in advance!

Jason

[Stephen_Sorkin]

This is a very interesting question and none of the answers actually get you to the full solution, where you find not only the top concurrency during the day, but when that occurred.

First you should note that the bin command simply flattens one field into "bins". For example, it turns seconds of the day into hours of the day. It alone cannot do the job here,it must be combined with concurrency to find the actual concurrency at any transition point in the time series (i.e., when a connection comes or goes).

Here's the first search, which will find the most recent occurrence of the top concurrency time of the day:

...
| eval duration = ReqTime/1000
| concurrency duration=duration
| bin span=dh _time as day
| dedup day sortby -concurrency
| eval when=_time
| timechart span=1d max(concurrency) as concurrency list(when) as when
| convert ctime(when)

Now your actual search is a bit trickier since we may have many points in the day of highest concurrency:

...
| eval duration = ReqTime/1000
| concurrency duration=duration
| bin span=1d _time as day
| eventstats max(concurrency) as max_concurrency by day
| eval when=if(concurrency==max_concurrency, _time, null())
| timechart span=1d max(concurrency) as concurrency list(when) as when
| convert ctime(when)

The general recipe here is:

1. Convert the duration to seconds.
2. Compute the level of concurrency at the start of every event.
3. Create a representation of the day of each request.
4. Either dedup to find the single highest concurrency point, or eventstats+eval to find the many highest equal concurrency points.
5. Use timechart to summarize the data by day.
6. Make the time of occurrence prettier.

Note that this technique can be used on splunk_access.log data using the following search:

index=_internal source=*/splunkd_access.log earliest=-24h@h
| eval duration = spent/1000
| concurrency duration=duration
| bin span=1h _time as hour
| eventstats max(concurrency) as max_concurrency by hour
| eval when=if(concurrency==max_concurrency, _time, null())
| timechart span=1h max(concurrency) as concurrency list(when) as when
| convert ctime(when)

[KangKangZhu]

Hi gkanapathy,

I just counted the requests by using grep from the log file directly, seems you are right the first expression wasn't give the right answer here, so i have two questions for you:

a).why the first one is wrong, i'm stupid and i use it was because i saw an example from the document

"Return the average "thruput" of each "host" for each 5 minute time span.
... | bucket _time span=5m | stats avg(thruput) by _time host"

so i think use it can put all events into discrete sets by using 1sec as time range, and then doing count for each set should get the right answer. ( i knew i didn't consider the ReqTime for calculating concurrency, however in my opinion, the result should match with the "grep")

b). If the second one is right, how can i set the range to 1 sec instead of 10 minutes

_time   max(concurrency)
1   5/16/10 5:**50**:00.000 PM  111
2   5/16/10 5:**40**:00.000 PM  141

and what's the unit of "duration"? the unit of ReqTime is msec in our system.

thanks

Jason

[gkanapathy]

although if it's long overall span, you might use |bucket _time span=1s | stats max(... instead of |timechart span=1s max(....

[gkanapathy]

if ReqTime is ms and not seconds, just do | eval ReqTimeSecs=ReqTime/1000 | concurrency duration=ReqTimeSecs. Duration, like _time and duration and most other times in Splunk, is in seconds.

I don't understand what the "range" you are looking for is. If it's just the highest value of "concurrency" in a given span, then use the "span=1s" parameter of "timechart".

[KangKangZhu]

Hi Guys,

Thanks for your quick responses!

I just got a chance to try "concurrency" in my local, however the result wasn't make sense to me, the case i was using for testing is

"Showing the max concurrency count in second level of day '05/16/2010'"

and i tried to use two expression to figure it out:

index=main Type=API earliest=05/16/2010:6:0:0 latest=05/16/2010:18:0:0 | bucket _time span=1s | stats count(_raw) by _time | sort -count(_raw) | head 1

_time count(_raw) 1 5/16/10 5:35:36.000 PM 28

index=main Type=API earliest=05/16/2010:6:0:0 latest=05/16/2010:18:0:0 | concurrency duration=ReqTime | timechart max(concurrency)

_time
max(concurrency) 1 5/16/10 6:00:00.000 AM 47

...

the result was different, i'm not sure which one is right, do you think the expression i used is make sense?

Thanks

Jason

[gkanapathy]

I don't see any reason to think that your first query would give you the right answer.

[sideview]

Indeed this is exactly the sort of use case that the concurrency command was added for:

<your search here> | concurrency duration=ReqTime | timechart max(concurrency)

Concurrency is only in 4.1. If you havent upgraded to 4.1 yet, it's possible there was a difficult and somewhat evil way of achieving the same thing with other search language and without concurrency but i dont know what it was.

http://www.splunk.com/base/Documentation/latest/SearchReference/Concurrency

[gkanapathy]

Seems to me you can use the concurrency search command: http://www.splunk.com/base/Documentation/latest/SearchReference/Concurrency