Getting Data In

delta report for multiple hosts

mcafeesecure
Explorer

I have the following log messages coming from syslog-ng

Jun 14 10:32:04 sc4-cron.mcafeesecure.com syslog-ng[2775]: Log statistics; dropped='tcp(AF_INET(10.98.202.126:514))=1613614', processed='center(queued)=175194', processed='center(received)=109152', processed='destination(d_resin)=51369', processed='destination(d_boot)=0', processed='destination(d_auth)=439', processed='destination(d_cron)=362', processed='destination(d_mlal)=0', processed='destination(loghost)=109152', processed='destination(d_mesg)=93', processed='destination(d_cons)=0', processed='destination(d_snmp)=6782', processed='destination(d_spol)=0', processed='destination(d_mail)=6997', processed='source(s_sys)=66165', processed='source(s_tail)=42187', processed='source(s_raccess)=800', processed='source(s_tail_2)=0'

Jun 14 10:22:04 sc4-cron.mcafeesecure.com syslog-ng[2775]: Log statistics; dropped='tcp(AF_INET(10.98.202.126:514))=1613614', processed='center(queued)=173017', processed='center(received)=107903', processed='destination(d_resin)=50855', processed='destination(d_boot)=0', processed='destination(d_auth)=430', processed='destination(d_cron)=359', processed='destination(d_mlal)=0', processed='destination(loghost)=107903', processed='destination(d_mesg)=92', processed='destination(d_cons)=0', processed='destination(d_snmp)=6610', processed='destination(d_spol)=0', processed='destination(d_mail)=6768', processed='source(s_sys)=65231', processed='source(s_tail)=41872', processed='source(s_raccess)=800', processed='source(s_tail_2)=0'

Jun 14 10:12:04 sc4-cron.mcafeesecure.com syslog-ng[2775]: Log statistics; dropped='tcp(AF_INET(10.98.202.126:514))=1613614', processed='center(queued)=171101', processed='center(received)=106828', processed='destination(d_resin)=50405', processed='destination(d_boot)=0', processed='destination(d_auth)=416', processed='destination(d_cron)=338', processed='destination(d_mlal)=0', processed='destination(loghost)=106828', processed='destination(d_mesg)=87', processed='destination(d_cons)=0', processed='destination(d_snmp)=6428', processed='destination(d_spol)=0', processed='destination(d_mail)=6599', processed='source(s_sys)=64382', processed='source(s_tail)=41646', processed='source(s_raccess)=800', processed='source(s_tail_2)=0'

I am using the following query to generate a diff of the field in this message I want (in this case the dropped field)

syslog-ng "Log statistics" host="sc4-cron*" | rex "(?i)^(?:[^=]=){2}(?P[^'])(?=')" | search DROPPED="" | delta DROPPED as DROPPEDDIFF | search DROPPEDDIFF=""

this works fine on a single host, however, I would like to run this across all of my hosts. The problem I then encounter, is that the previous log message is not from the same host as the latest message, therefore the diffs that are created are not accurate.

I have hundreds of hosts, and would like to build a report on this data, however, I can't really go and do them all individually. In addition getting all of this into one report would be ideal.

Tags (2)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

You can use the streamstats command, something like:

... | streamstats current=f global=f window=1 first(DROPPED) as NEXT_DROPPED by host | eval DROPPEDDIFF = NEXT_DROPPED-DROPPED

Something like that. You may have to reverse the subtraction, I'm not sure which way around it goes, and well, it might be off-by-one, i.e. the diff might be on the previous vs the current result, but you get the idea.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

You can use the streamstats command, something like:

... | streamstats current=f global=f window=1 first(DROPPED) as NEXT_DROPPED by host | eval DROPPEDDIFF = NEXT_DROPPED-DROPPED

Something like that. You may have to reverse the subtraction, I'm not sure which way around it goes, and well, it might be off-by-one, i.e. the diff might be on the previous vs the current result, but you get the idea.

mcafeesecure
Explorer

Thanks.
For the record, the final query looks like this:

syslog-ng "Log statistics" host!="qa2-cron*" | rex "(?i)^(?:[^=]=){2}(?P[^'])(?=')" | search DROPPED="*" | streamstats current=f global=f window=1 first(DROPPED) as NEXT_DROPPED by host | eval DROPPEDDIFF = NEXT_DROPPED-DROPPED | search DROPPEDDIFF>0

morian
Engager

Thank you, you helped me solve my problem.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

.conf25 Global Broadcast: Don’t Miss a Moment

Hello Splunkers, .conf25 is only a click away.  Not able to make it to .conf25 in person? No worries, you can ...

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...