Getting Data In

delta report for multiple hosts

mcafeesecure
Explorer

I have the following log messages coming from syslog-ng

Jun 14 10:32:04 sc4-cron.mcafeesecure.com syslog-ng[2775]: Log statistics; dropped='tcp(AF_INET(10.98.202.126:514))=1613614', processed='center(queued)=175194', processed='center(received)=109152', processed='destination(d_resin)=51369', processed='destination(d_boot)=0', processed='destination(d_auth)=439', processed='destination(d_cron)=362', processed='destination(d_mlal)=0', processed='destination(loghost)=109152', processed='destination(d_mesg)=93', processed='destination(d_cons)=0', processed='destination(d_snmp)=6782', processed='destination(d_spol)=0', processed='destination(d_mail)=6997', processed='source(s_sys)=66165', processed='source(s_tail)=42187', processed='source(s_raccess)=800', processed='source(s_tail_2)=0'

Jun 14 10:22:04 sc4-cron.mcafeesecure.com syslog-ng[2775]: Log statistics; dropped='tcp(AF_INET(10.98.202.126:514))=1613614', processed='center(queued)=173017', processed='center(received)=107903', processed='destination(d_resin)=50855', processed='destination(d_boot)=0', processed='destination(d_auth)=430', processed='destination(d_cron)=359', processed='destination(d_mlal)=0', processed='destination(loghost)=107903', processed='destination(d_mesg)=92', processed='destination(d_cons)=0', processed='destination(d_snmp)=6610', processed='destination(d_spol)=0', processed='destination(d_mail)=6768', processed='source(s_sys)=65231', processed='source(s_tail)=41872', processed='source(s_raccess)=800', processed='source(s_tail_2)=0'

Jun 14 10:12:04 sc4-cron.mcafeesecure.com syslog-ng[2775]: Log statistics; dropped='tcp(AF_INET(10.98.202.126:514))=1613614', processed='center(queued)=171101', processed='center(received)=106828', processed='destination(d_resin)=50405', processed='destination(d_boot)=0', processed='destination(d_auth)=416', processed='destination(d_cron)=338', processed='destination(d_mlal)=0', processed='destination(loghost)=106828', processed='destination(d_mesg)=87', processed='destination(d_cons)=0', processed='destination(d_snmp)=6428', processed='destination(d_spol)=0', processed='destination(d_mail)=6599', processed='source(s_sys)=64382', processed='source(s_tail)=41646', processed='source(s_raccess)=800', processed='source(s_tail_2)=0'

I am using the following query to generate a diff of the field in this message I want (in this case the dropped field)

syslog-ng "Log statistics" host="sc4-cron*" | rex "(?i)^(?:[^=]=){2}(?P[^'])(?=')" | search DROPPED="" | delta DROPPED as DROPPEDDIFF | search DROPPEDDIFF=""

this works fine on a single host, however, I would like to run this across all of my hosts. The problem I then encounter, is that the previous log message is not from the same host as the latest message, therefore the diffs that are created are not accurate.

I have hundreds of hosts, and would like to build a report on this data, however, I can't really go and do them all individually. In addition getting all of this into one report would be ideal.

Tags (2)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

You can use the streamstats command, something like:

... | streamstats current=f global=f window=1 first(DROPPED) as NEXT_DROPPED by host | eval DROPPEDDIFF = NEXT_DROPPED-DROPPED

Something like that. You may have to reverse the subtraction, I'm not sure which way around it goes, and well, it might be off-by-one, i.e. the diff might be on the previous vs the current result, but you get the idea.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

You can use the streamstats command, something like:

... | streamstats current=f global=f window=1 first(DROPPED) as NEXT_DROPPED by host | eval DROPPEDDIFF = NEXT_DROPPED-DROPPED

Something like that. You may have to reverse the subtraction, I'm not sure which way around it goes, and well, it might be off-by-one, i.e. the diff might be on the previous vs the current result, but you get the idea.

mcafeesecure
Explorer

Thanks.
For the record, the final query looks like this:

syslog-ng "Log statistics" host!="qa2-cron*" | rex "(?i)^(?:[^=]=){2}(?P[^'])(?=')" | search DROPPED="*" | streamstats current=f global=f window=1 first(DROPPED) as NEXT_DROPPED by host | eval DROPPEDDIFF = NEXT_DROPPED-DROPPED | search DROPPEDDIFF>0

morian
Engager

Thank you, you helped me solve my problem.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...

Customer success is front and center at .conf25

Hi Splunkers, If you are not able to be at .conf25 in person, you can still learn about all the latest news ...