Extracting multiple occurrences of a field from a syslog-ng statistics message.

[brettw10]

Hi,

I am trying to extract multiple occurrences of two fields from the statistics message that is generated by syslog-ng. You can find an example single-lined entry below. I need to extract every destination name - between dropped='dst.tcp( and # - and the corresponding dropped statistic value - between )= and '.

I tried adding the following to props and transforms, but haven't had any luck so far:

props.conf

[syslog-ng_syslog]
KV_MODE = auto
REPORT-syslog_host = syslog-ng_host
REPORT-syslog_host_drops = syslog-ng_drops

transforms.conf

[syslog-ng_host]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped='dst.tcp[(](?<syslog_host>\w+[^#]+)
SOURCE_KEY = _raw

[syslog-ng_drops]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped='dst.tcp[(]\S+[)]=(?<syslog_drops>\d+[^'])
SOURCE_KEY = _raw

Sample statistics entry

Feb 13 03:11:01 slavvwelss01v01 syslog-ng[19001]: Log statistics; processed='destination(d_10.173.249.211_5)=100544707', processed='source(s_network_6)=100544708', processed='global(payload_reallocs)=401769', dropped='dst.tcp(<b>d_10.173.249.211_3</b>#0,10.173.249.211:514)=<b>88484837</b>', processed='dst.tcp(d_10.173.249.211_3#0,10.173.249.211:514)=100544708', stored='dst.tcp(d_10.173.249.211_3#0,10.173.249.211:514)=20001', processed='destination(d_10.173.249.211_6)=100544708', processed='source(s_network_7)=100544560', dropped='dst.tcp(<b>d_10.173.249.211_6</b>#0,10.173.249.211:514)=<b>88263146</b>', processed='dst.tcp(d_10.173.249.211_6#0,10.173.249.211:514)=100544709', stored='dst.tcp(d_10.173.249.211_6#0,10.173.249.211:514)=20001', processed='destination(d_10.173.249.211_7)=100544560', processed='source(s_local)=225508438', processed='global(msg_clones)=0', dropped='dst.tcp(d_10.173.249.211_1#0,10.173.249.211:514)=300011709', processed='dst.tcp(d_10.173.249.211_1#0,10.173.249.211:514)=326053162', stored='dst.tcp(d_10.173.249.211_1#0,10.173.249.211:514)=20001', processed='destination(d_local_msgs)=225508438', dropped='dst.tcp(d_10.173.249.211_4#0,10.173.249.211:514)=88282600', processed='dst.tcp(d_10.173.249.211_4#0,10.173.249.211:514)=100544706', stored='dst.tcp(d_10.173.249.211_4#0,10.173.249.211:514)=20001', processed='src.internal(s_local#1)=225506018', stamp='src.internal(s_local#1)=1360685461', processed='source(s_network_1)=100544726', dropped='dst.tcp(d_10.173.249.211_7#0,10.173.249.211:514)=88356001', processed='dst.tcp(d_10.173.249.211_7#0,10.173.249.211:514)=100544562', stored='dst.tcp(d_10.173.249.211_7#0,10.173.249.211:514)=20001', processed='global(sdata_updates)=1407495255', processed='destination(d_10.173.249.211_1)=326053164', processed='source(s_network_2)=100544585', dropped='dst.tcp(d_10.173.249.211_2#0,10.173.249.211:514)=88193930', processed='dst.tcp(d_10.173.249.211_2#0,10.173.249.211:514)=100544585', stored='dst.tcp(d_10.173.249.211_2#0,10.173.249.211:514)=20001', processed='destination(d_10.173.249.211_2)=100544585', processed='source(s_network_3)=100544708', processed='center(received)=0', processed='destination(d_10.173.249.211_3)=100544708', processed='source(s_network_4)=100544706', dropped='dst.tcp(d_10.173.249.211_5#0,10.173.249.211:514)=88461990', processed='dst.tcp(d_10.173.249.211_5#0,10.173.249.211:514)=100544707', stored='dst.tcp(d_10.173.249.211_5#0,10.173.249.211:514)=20001', processed='center(queued)=0', processed='destination(d_10.173.249.211_4)=100544706', processed='source(s_network_5)=100544707'

Any help would be greatly appreciated.

Regards,
Brett.

[brettw10 · Accepted solution]

Hi,

It actually turns out that I had forgotten to set the right sourcetype in props.conf. Here is the correct props.conf:

[syslog]
KV_MODE = auto
REPORT-syslog_host = syslog-ng_host
REPORT-syslog_host_drops = syslog-ng_drops

Rgds,
Brett.

[lguinn2]

Try this:

[syslog-ng_host]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped=\'dst.tcp\((?:\<b\>)*(?<syslog_host>\S+?(?:\</b\>)*\#)
SOURCE_KEY = _raw

[syslog-ng_drops]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped=\'dst.tcp.*?=(?<syslog_drops>\d+)\'
SOURCE_KEY = _raw