index esxi logs on central syslog-ng/splunk host

[mettlers]

Hi there

we're running a central syslog-ng host where we collect all logs relevant to us. usually delivered by syslog-ng agents on the respecting source hosts, in case of our esxi servers via standard syslog relayed through a syslog-ng "forwarder" in the same admin vlan.

esxi hostname "extraction" works fine, we write the files into a structure like
../vmware/[esxihostname]/vmware.log-20110708
-> splunk data input with host = segment in path.

my problem is that esxi multiline events occurring regularly are not recognized as such because syslog-ng adds its own timestamp in front of every line.

any hints on how to solve this are greatly appreciated!

regards

[splunk24]

Can you please let me know how you are gettting the vmware.log from syslog .. i think vmware esxi host is not sending the vmware.log as such through syslog..
what changes you have made in syslog to get vmware.log

[yannK]

The issue here is that you are sending non syslog data to a syslog server, and after are indexing the result therefore they are indexed with the sourcetype=syslog, and parsed as single line.

There are methods to create a new sourcetype, and change the parsing rules, but your event will still be polluted by the timestamp/host at each line. (or you have to play with the sedcmd command and actually remove some parts of the events, but it may remove the timestamp and the host information)

Here is a way to break the events correctly :

• see details here http://www.splunk.com/base/Documentation/latest/Data/Indexmulti-lineevents
• the timestamp used will be the one of the esxi event, not the one from the syslog-ng. You can change it using http://www.splunk.com/base/Documentation/latest/Data/Configuretimestamprecognition
• the linebreaking is the line with "date host [date "
• for the host extraction we are just calling the existing transforms defined for syslog

inputs.conf

[monitor://mylogpath/myesxi.log]

sourcetype=esxi

props.conf

[esxi]

SHOULD_LINEMERGE = true

BREAK_ONLY_BEFORE_DATE = false

LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{2}:\d{2} [^\s].* [\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}

TRANSFORMS-esxihost=syslog-host

[mettlers]

no one?

logs to index by splunk look like this i might add

2011-07-11T00:09:59+02:00 host4 [2011-07-11 00:09:59.093 1972DB90 verbose 'App'] [VpxaVMAP::Invoke] Command output:
2011-07-11T02:09:59+02:00 host4  -z -shortname=host4 -uname=VMkernel -cmd=monitornodes -domain=vmware
2011-07-11T02:09:59+02:00 host4      FT_ISOLATION_TIME=1
2011-07-11T02:09:59+02:00 host4 09:58 [print_args          ]      LD_LIBRARY_PATH=/lib:/usr/lib:/opt/vmware/aam/lib:/opt/vmware/vpxa/vpx:
2011-07-11T02:09:59+02:00 host4 00:09:58 [print_args          ]      PWD=/var/log/vmware/vpx
2011-07-11T02:09:59+02:00 host4 /usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin
2011-07-11T02:09:59+02:00 host4 00:09:58 [print_args          ]      cmd=monitornodes
2011-07-11T02:09:59+02:00 host4 58 [print_args          ]      domain=vmware
2011-07-11T02:09:59+02:00 host4
2011-07-11T02:09:59+02:00 host4 CMD:    /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes
2011-07-11T02:09:59+02:00 host4 the master primary ***
2011-07-11T02:09:59+02:00 host4   host4                 Primary      Agent Running
2011-07-11T02:09:59+02:00 host4 58 [issue_cmd           ]   hvmc43                 Primary      Agent Running
2011-07-11T02:09:59+02:00 host4 00:09:58 [issue_cmd           ] CMD:    /bin/ping -c 1 192.168.0.254
2011-07-11T02:09:59+02:00 host4 56 data bytes
2011-07-11T02:09:59+02:00 host4 09:58 [issue_cmd           ] 1 packets transmitted, 1 packets received, 0% packet loss
2011-07-11T02:09:59+02:00 host4 VMwareresult=success
2011-07-11T00:09:59+02:00 host4
2011-07-11T00:09:59+02:00 host4 [2011-07-11 00:09:59.093 1972DB90 verbose 'App'] [VpxaVMAP::Invoke] Command returned successfully
2011-07-11T00:09:59+02:00 host4 [2011-07-11 00:09:59.755 195A7B90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'

Splunk indexes each line as one event (as expected). there are only 3 esxi events here though, starting with syslog-ng_timestamp host4 [2011-07-11] 00:09:59....

any ideas on how to take esxi's timestamp as separators withouth changing the syslog-ng config (if possible at all) or using splunk forwarder etc.?