topic Re: Splunk for mathematical difference in stats? in Splunk Search

How to use Splunk for mathematical difference in stats?

chengka — Thu, 24 Feb 2022 18:50:16 GMT

Hello,
I would like to try using Splunk to calculate the difference in numbers from one sample to the next. Here is some theoretical log entries.

The data indexed will look like this:

yesterday_timeStamp clusterId=abc, topicName=mytopic, partition=0, lastOffset=100 yesterday_timeStamp clusterId=abc, topicName=mytopic, partition=1, lastOffset=200 today_timeStamp clusterId=abc, topicName=mytopic, partition=0, lastOffset=355 today_timeStamp clusterId=abc, topicName=mytopic, partition=1, lastOffset=401

The number of events in the last 24 hours would be partition 0(355-200=155), partition 1(401-200=201) Sum of partitions for topic(mytopic) = 155+201=356
There will be many topicName(s) and possible different numbers of partition(s) per topicName.
Can I use splunk to calculate the numbers of events per partition and topic since yesterday?

Re: Splunk for mathematical difference in stats?

ITWhisperer — Mon, 07 Feb 2022 18:38:25 GMT

| streamstats window=2 global=f range(lastOffset) as difference by clusterId topicName partition | where timestamp="today_timeStamp" | stats sum(difference) as events by clusterId topicName

Re: Splunk for mathematical difference in stats?

chengka — Thu, 24 Feb 2022 18:00:09 GMT

Thank you I appreciate the help. I was too literal. The timestamp will be a real date/time. How do I structure this where clause to be the latest?(all events will present the same date/time for each "interval", as described that would be a day)

| where timestamp="today_timeStamp"

Re: Splunk for mathematical difference in stats?

yuanliu — Fri, 25 Feb 2022 09:24:34 GMT

I still don't see why ITWhisperer's tstats formula should not work, but simple stats should also be able to do the job assuming the "since yesterday" piece is done with search time filter.

| stats max(lastOffset) as maxOffset min(lastOffset) as minOffset by partition topicName clusterId | eval numEvents=maxOffset - minOffset | eventstats sum(numEvents) as totalEvents

One thing I can't figure out is how to derive number of events in partition 0 as 355-200=155; shouldn't it be 355-100=255? (Lines one and three.) Is there a relation that I missed?

The above gives

patition	topicName	clusterId	maxOffset	minOffset	numEvents	totalEvents
0	mytopic	abc	355	100	255	456
1	mytopic	abc	401	200	201	456

Re: Splunk for mathematical difference in stats?

chengka — Fri, 25 Feb 2022 15:14:13 GMT

Thank you. You are correct, 355-200 was a mistake.