topic Re: delta counts by keyname in Splunk Search

delta counts by keyname

malex — Fri, 24 Jan 2014 18:14:23 GMT

How can I get a delta count by a key name when there are multiple keys for plotting the delta in a report?

I have a collection that outputs like this via syslog:

TimeStampMsec="1390586680463" QueueName="ad.input" ConsumerCount="1" MessagePendingCount="0" EnqueueCount="9" DequeueCount="9"
TimeStampMsec="1390586680463" QueueName="ldap.input" ConsumerCount="0" MessagePendingCount="0" EnqueueCount="0" DequeueCount="0"
TimeStampMsec="1390586680463" QueueName="foo.bar" ConsumerCount="0" MessagePendingCount="4" EnqueueCount="0" DequeueCount="0"

The DequeueCount could increment for the next log entry for any of these records as identified by the QueueName key. I would like to setup a report that provides a linear graph by time for by QueueName of the delta on DequeueCount. I cannot figure this out with delta since I can't seem to get it to take the delta by the QueueName, it can only take the delta of the record previously.

I have done this with mvlist, but we could add/subtract the QueueNames and mvlist feels like it's accessing points via an array and I can't guarantee the order.

Re: delta counts by keyname

martin_mueller — Fri, 24 Jan 2014 23:43:37 GMT

To compute a grouped-by delta you can use streamstats:

base search | streamstats window=1 current=f global=f last(value) as last by key | eval delta = value-last | charting stuff

(precise syntax may vary)

Re: delta counts by keyname

kristian_kolb — Fri, 24 Jan 2014 23:47:43 GMT

Maybe not perfect, but can perhaps serve as some inspiration

your_base_search 
| sort QueueName, _time 
| streamstats window=1 current=f first(DequeueCount) AS prevDQ by QueueName 
| table _time QueueName DequeueCount prevDQ  
| eval delta = DequeueCount - prevDQ 
| fields - prevDQ

Re: delta counts by keyname

kristian_kolb — Fri, 24 Jan 2014 23:48:28 GMT

dammit, spent too long editing 🙂

Re: delta counts by keyname

kristian_kolb — Fri, 24 Jan 2014 23:51:00 GMT

but you'll need to sort before the streamstats

Re: delta counts by keyname

martin_mueller — Sat, 25 Jan 2014 00:00:55 GMT

Sure, but I assume that's done by the base search 😛 else the delta without grouping wouldn't have worked properly either.

Re: delta counts by keyname

alacercogitatus — Sat, 25 Jan 2014 00:02:40 GMT

This is EXTREMELY EXPERIMENTAL - so your feedback is appreciated. It has been submitted to SplunkBase - hasn't been approved yet - feel free to try it out. But Seriously - I only tested this on a 6.0 Linux install. Use At YOUR OWN RISK 😄 (But send me feedback if it craps.)

http://data.kyleasmith.info/TA-deltaby.spl

The command for your data would look like this:

your_base_search | deltaby by=QueueName value=DequeueCount output=DeQueueCountDelta | timechart span=10m sum(DeQueueCountDelta) by QueueName

output is optional but useful. I think you can even stack the commands (I haven't tested that yet fully)

your_base_search | deltaby by=QueueName value=DequeueCount output=DequeueDelta | deltaby=QueueName value=EnqueueCount output=EnqueueDelta | timechart span=10m sum(DequeueDelta) sum(EnqueueDelta) by QueueName

Again: Experimental (v0.1). Prone to failure (depending on your inputs). Send me feedback: splunkapps@kyleasmith.info.

Re: delta counts by keyname

malex — Sat, 25 Jan 2014 01:43:02 GMT

This is getting there, thanks a lot.