Getting Data In

Extracting multiple occurrences of a field from a syslog-ng statistics message.

brettw10
Explorer

Hi,

I am trying to extract multiple occurrences of two fields from the statistics message that is generated by syslog-ng. You can find an example single-lined entry below. I need to extract every destination name - between dropped='dst.tcp( and # - and the corresponding dropped statistic value - between )= and '.

I tried adding the following to props and transforms, but haven't had any luck so far:

props.conf

[syslog-ng_syslog]
KV_MODE = auto
REPORT-syslog_host = syslog-ng_host
REPORT-syslog_host_drops = syslog-ng_drops

transforms.conf

[syslog-ng_host]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped='dst.tcp[(](?<syslog_host>\w+[^#]+)
SOURCE_KEY = _raw

[syslog-ng_drops]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped='dst.tcp[(]\S+[)]=(?<syslog_drops>\d+[^'])
SOURCE_KEY = _raw

Sample statistics entry

Feb 13 03:11:01 slavvwelss01v01 syslog-ng[19001]: Log statistics; processed='destination(d_10.173.249.211_5)=100544707', processed='source(s_network_6)=100544708', processed='global(payload_reallocs)=401769', dropped='dst.tcp(<b>d_10.173.249.211_3</b>#0,10.173.249.211:514)=<b>88484837</b>', processed='dst.tcp(d_10.173.249.211_3#0,10.173.249.211:514)=100544708', stored='dst.tcp(d_10.173.249.211_3#0,10.173.249.211:514)=20001', processed='destination(d_10.173.249.211_6)=100544708', processed='source(s_network_7)=100544560', dropped='dst.tcp(<b>d_10.173.249.211_6</b>#0,10.173.249.211:514)=<b>88263146</b>', processed='dst.tcp(d_10.173.249.211_6#0,10.173.249.211:514)=100544709', stored='dst.tcp(d_10.173.249.211_6#0,10.173.249.211:514)=20001', processed='destination(d_10.173.249.211_7)=100544560', processed='source(s_local)=225508438', processed='global(msg_clones)=0', dropped='dst.tcp(d_10.173.249.211_1#0,10.173.249.211:514)=300011709', processed='dst.tcp(d_10.173.249.211_1#0,10.173.249.211:514)=326053162', stored='dst.tcp(d_10.173.249.211_1#0,10.173.249.211:514)=20001', processed='destination(d_local_msgs)=225508438', dropped='dst.tcp(d_10.173.249.211_4#0,10.173.249.211:514)=88282600', processed='dst.tcp(d_10.173.249.211_4#0,10.173.249.211:514)=100544706', stored='dst.tcp(d_10.173.249.211_4#0,10.173.249.211:514)=20001', processed='src.internal(s_local#1)=225506018', stamp='src.internal(s_local#1)=1360685461', processed='source(s_network_1)=100544726', dropped='dst.tcp(d_10.173.249.211_7#0,10.173.249.211:514)=88356001', processed='dst.tcp(d_10.173.249.211_7#0,10.173.249.211:514)=100544562', stored='dst.tcp(d_10.173.249.211_7#0,10.173.249.211:514)=20001', processed='global(sdata_updates)=1407495255', processed='destination(d_10.173.249.211_1)=326053164', processed='source(s_network_2)=100544585', dropped='dst.tcp(d_10.173.249.211_2#0,10.173.249.211:514)=88193930', processed='dst.tcp(d_10.173.249.211_2#0,10.173.249.211:514)=100544585', stored='dst.tcp(d_10.173.249.211_2#0,10.173.249.211:514)=20001', processed='destination(d_10.173.249.211_2)=100544585', processed='source(s_network_3)=100544708', processed='center(received)=0', processed='destination(d_10.173.249.211_3)=100544708', processed='source(s_network_4)=100544706', dropped='dst.tcp(d_10.173.249.211_5#0,10.173.249.211:514)=88461990', processed='dst.tcp(d_10.173.249.211_5#0,10.173.249.211:514)=100544707', stored='dst.tcp(d_10.173.249.211_5#0,10.173.249.211:514)=20001', processed='center(queued)=0', processed='destination(d_10.173.249.211_4)=100544706', processed='source(s_network_5)=100544707'

Any help would be greatly appreciated.

Regards,
Brett.

0 Karma
1 Solution

brettw10
Explorer

Hi,

It actually turns out that I had forgotten to set the right sourcetype in props.conf. Here is the correct props.conf:

[syslog]
KV_MODE = auto
REPORT-syslog_host = syslog-ng_host
REPORT-syslog_host_drops = syslog-ng_drops

Rgds,
Brett.

View solution in original post

0 Karma

brettw10
Explorer

Hi,

It actually turns out that I had forgotten to set the right sourcetype in props.conf. Here is the correct props.conf:

[syslog]
KV_MODE = auto
REPORT-syslog_host = syslog-ng_host
REPORT-syslog_host_drops = syslog-ng_drops

Rgds,
Brett.

0 Karma

lguinn2
Legend

Try this:

[syslog-ng_host]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped=\'dst.tcp\((?:\<b\>)*(?<syslog_host>\S+?(?:\</b\>)*\#)
SOURCE_KEY = _raw

[syslog-ng_drops]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped=\'dst.tcp.*?=(?<syslog_drops>\d+)\'
SOURCE_KEY = _raw
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...