How can I get a delta count by a key name when there are multiple keys for plotting the delta in a report?
I have a collection that outputs like this via syslog:
TimeStampMsec="1390586680463" QueueName="ad.input" ConsumerCount="1" MessagePendingCount="0" EnqueueCount="9" DequeueCount="9"
TimeStampMsec="1390586680463" QueueName="ldap.input" ConsumerCount="0" MessagePendingCount="0" EnqueueCount="0" DequeueCount="0"
TimeStampMsec="1390586680463" QueueName="foo.bar" ConsumerCount="0" MessagePendingCount="4" EnqueueCount="0" DequeueCount="0"
The DequeueCount could increment for the next log entry for any of these records as identified by the QueueName key. I would like to setup a report that provides a linear graph by time for by QueueName of the delta on DequeueCount. I cannot figure this out with delta since I can't seem to get it to take the delta by the QueueName, it can only take the delta of the record previously.
I have done this with mvlist, but we could add/subtract the QueueNames and mvlist feels like it's accessing points via an array and I can't guarantee the order.
Maybe not perfect, but can perhaps serve as some inspiration
your_base_search
| sort QueueName, _time
| streamstats window=1 current=f first(DequeueCount) AS prevDQ by QueueName
| table _time QueueName DequeueCount prevDQ
| eval delta = DequeueCount - prevDQ
| fields - prevDQ
/K
This is EXTREMELY EXPERIMENTAL - so your feedback is appreciated. It has been submitted to SplunkBase - hasn't been approved yet - feel free to try it out. But Seriously - I only tested this on a 6.0 Linux install. Use At YOUR OWN RISK 😄 (But send me feedback if it craps.)
http://data.kyleasmith.info/TA-deltaby.spl
The command for your data would look like this:
your_base_search | deltaby by=QueueName value=DequeueCount output=DeQueueCountDelta | timechart span=10m sum(DeQueueCountDelta) by QueueName
output is optional but useful. I think you can even stack the commands (I haven't tested that yet fully)
your_base_search | deltaby by=QueueName value=DequeueCount output=DequeueDelta | deltaby=QueueName value=EnqueueCount output=EnqueueDelta | timechart span=10m sum(DequeueDelta) sum(EnqueueDelta) by QueueName
Again: Experimental (v0.1). Prone to failure (depending on your inputs). Send me feedback: splunkapps@kyleasmith.info.
Maybe not perfect, but can perhaps serve as some inspiration
your_base_search
| sort QueueName, _time
| streamstats window=1 current=f first(DequeueCount) AS prevDQ by QueueName
| table _time QueueName DequeueCount prevDQ
| eval delta = DequeueCount - prevDQ
| fields - prevDQ
/K
This is getting there, thanks a lot.
To compute a grouped-by delta you can use streamstats
:
base search | streamstats window=1 current=f global=f last(value) as last by key | eval delta = value-last | charting stuff
(precise syntax may vary)
Sure, but I assume that's done by the base search 😛 else the delta without grouping wouldn't have worked properly either.
but you'll need to sort before the streamstats
dammit, spent too long editing 🙂