Getting Data In

Extracting multiple occurrences of a field from a syslog-ng statistics message.

Explorer

Hi,

I am trying to extract multiple occurrences of two fields from the statistics message that is generated by syslog-ng. You can find an example single-lined entry below. I need to extract every destination name - between dropped='dst.tcp( and # - and the corresponding dropped statistic value - between )= and '.

I tried adding the following to props and transforms, but haven't had any luck so far:

props.conf

[syslog-ng_syslog]
KV_MODE = auto
REPORT-syslog_host = syslog-ng_host
REPORT-syslog_host_drops = syslog-ng_drops

transforms.conf

[syslog-ng_host]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped='dst.tcp[(](?<syslog_host>\w+[^#]+)
SOURCE_KEY = _raw

[syslog-ng_drops]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped='dst.tcp[(]\S+[)]=(?<syslog_drops>\d+[^'])
SOURCE_KEY = _raw

Sample statistics entry

Feb 13 03:11:01 slavvwelss01v01 syslog-ng[19001]: Log statistics; processed='destination(d_10.173.249.211_5)=100544707', processed='source(s_network_6)=100544708', processed='global(payload_reallocs)=401769', dropped='dst.tcp(<b>d_10.173.249.211_3</b>#0,10.173.249.211:514)=<b>88484837</b>', processed='dst.tcp(d_10.173.249.211_3#0,10.173.249.211:514)=100544708', stored='dst.tcp(d_10.173.249.211_3#0,10.173.249.211:514)=20001', processed='destination(d_10.173.249.211_6)=100544708', processed='source(s_network_7)=100544560', dropped='dst.tcp(<b>d_10.173.249.211_6</b>#0,10.173.249.211:514)=<b>88263146</b>', processed='dst.tcp(d_10.173.249.211_6#0,10.173.249.211:514)=100544709', stored='dst.tcp(d_10.173.249.211_6#0,10.173.249.211:514)=20001', processed='destination(d_10.173.249.211_7)=100544560', processed='source(s_local)=225508438', processed='global(msg_clones)=0', dropped='dst.tcp(d_10.173.249.211_1#0,10.173.249.211:514)=300011709', processed='dst.tcp(d_10.173.249.211_1#0,10.173.249.211:514)=326053162', stored='dst.tcp(d_10.173.249.211_1#0,10.173.249.211:514)=20001', processed='destination(d_local_msgs)=225508438', dropped='dst.tcp(d_10.173.249.211_4#0,10.173.249.211:514)=88282600', processed='dst.tcp(d_10.173.249.211_4#0,10.173.249.211:514)=100544706', stored='dst.tcp(d_10.173.249.211_4#0,10.173.249.211:514)=20001', processed='src.internal(s_local#1)=225506018', stamp='src.internal(s_local#1)=1360685461', processed='source(s_network_1)=100544726', dropped='dst.tcp(d_10.173.249.211_7#0,10.173.249.211:514)=88356001', processed='dst.tcp(d_10.173.249.211_7#0,10.173.249.211:514)=100544562', stored='dst.tcp(d_10.173.249.211_7#0,10.173.249.211:514)=20001', processed='global(sdata_updates)=1407495255', processed='destination(d_10.173.249.211_1)=326053164', processed='source(s_network_2)=100544585', dropped='dst.tcp(d_10.173.249.211_2#0,10.173.249.211:514)=88193930', processed='dst.tcp(d_10.173.249.211_2#0,10.173.249.211:514)=100544585', stored='dst.tcp(d_10.173.249.211_2#0,10.173.249.211:514)=20001', processed='destination(d_10.173.249.211_2)=100544585', processed='source(s_network_3)=100544708', processed='center(received)=0', processed='destination(d_10.173.249.211_3)=100544708', processed='source(s_network_4)=100544706', dropped='dst.tcp(d_10.173.249.211_5#0,10.173.249.211:514)=88461990', processed='dst.tcp(d_10.173.249.211_5#0,10.173.249.211:514)=100544707', stored='dst.tcp(d_10.173.249.211_5#0,10.173.249.211:514)=20001', processed='center(queued)=0', processed='destination(d_10.173.249.211_4)=100544706', processed='source(s_network_5)=100544707'

Any help would be greatly appreciated.

Regards,
Brett.

0 Karma
1 Solution

Explorer

Hi,

It actually turns out that I had forgotten to set the right sourcetype in props.conf. Here is the correct props.conf:

[syslog]
KV_MODE = auto
REPORT-syslog_host = syslog-ng_host
REPORT-syslog_host_drops = syslog-ng_drops

Rgds,
Brett.

View solution in original post

0 Karma

Explorer

Hi,

It actually turns out that I had forgotten to set the right sourcetype in props.conf. Here is the correct props.conf:

[syslog]
KV_MODE = auto
REPORT-syslog_host = syslog-ng_host
REPORT-syslog_host_drops = syslog-ng_drops

Rgds,
Brett.

View solution in original post

0 Karma

Legend

Try this:

[syslog-ng_host]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped=\'dst.tcp\((?:\<b\>)*(?<syslog_host>\S+?(?:\</b\>)*\#)
SOURCE_KEY = _raw

[syslog-ng_drops]
CLEAN_KEYS = 1
MV_ADD = 1
REGEX = dropped=\'dst.tcp.*?=(?<syslog_drops>\d+)\'
SOURCE_KEY = _raw
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!