Getting Data In

index esxi logs on central syslog-ng/splunk host


Hi there

we're running a central syslog-ng host where we collect all logs relevant to us. usually delivered by syslog-ng agents on the respecting source hosts, in case of our esxi servers via standard syslog relayed through a syslog-ng "forwarder" in the same admin vlan.

esxi hostname "extraction" works fine, we write the files into a structure like
-> splunk data input with host = segment in path.

my problem is that esxi multiline events occurring regularly are not recognized as such because syslog-ng adds its own timestamp in front of every line.

any hints on how to solve this are greatly appreciated!


Tags (3)

Path Finder

Can you please let me know how you are gettting the vmware.log from syslog .. i think vmware esxi host is not sending the vmware.log as such through syslog..
what changes you have made in syslog to get vmware.log

0 Karma

Splunk Employee
Splunk Employee

The issue here is that you are sending non syslog data to a syslog server, and after are indexing the result therefore they are indexed with the sourcetype=syslog, and parsed as single line.

There are methods to create a new sourcetype, and change the parsing rules, but your event will still be polluted by the timestamp/host at each line. (or you have to play with the sedcmd command and actually remove some parts of the events, but it may remove the timestamp and the host information)

Here is a way to break the events correctly :




LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{2}:\d{2} [^\s].* [\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}

0 Karma


no one?

logs to index by splunk look like this i might add

2011-07-11T00:09:59+02:00 host4 [2011-07-11 00:09:59.093 1972DB90 verbose 'App'] [VpxaVMAP::Invoke] Command output:
2011-07-11T02:09:59+02:00 host4  -z -shortname=host4 -uname=VMkernel -cmd=monitornodes -domain=vmware
2011-07-11T02:09:59+02:00 host4      FT_ISOLATION_TIME=1
2011-07-11T02:09:59+02:00 host4 09:58 [print_args          ]      LD_LIBRARY_PATH=/lib:/usr/lib:/opt/vmware/aam/lib:/opt/vmware/vpxa/vpx:
2011-07-11T02:09:59+02:00 host4 00:09:58 [print_args          ]      PWD=/var/log/vmware/vpx
2011-07-11T02:09:59+02:00 host4 /usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin
2011-07-11T02:09:59+02:00 host4 00:09:58 [print_args          ]      cmd=monitornodes
2011-07-11T02:09:59+02:00 host4 58 [print_args          ]      domain=vmware
2011-07-11T02:09:59+02:00 host4
2011-07-11T02:09:59+02:00 host4 CMD:    /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes
2011-07-11T02:09:59+02:00 host4 the master primary ***
2011-07-11T02:09:59+02:00 host4   host4                 Primary      Agent Running
2011-07-11T02:09:59+02:00 host4 58 [issue_cmd           ]   hvmc43                 Primary      Agent Running
2011-07-11T02:09:59+02:00 host4 00:09:58 [issue_cmd           ] CMD:    /bin/ping -c 1
2011-07-11T02:09:59+02:00 host4 56 data bytes
2011-07-11T02:09:59+02:00 host4 09:58 [issue_cmd           ] 1 packets transmitted, 1 packets received, 0% packet loss
2011-07-11T02:09:59+02:00 host4 VMwareresult=success
2011-07-11T00:09:59+02:00 host4
2011-07-11T00:09:59+02:00 host4 [2011-07-11 00:09:59.093 1972DB90 verbose 'App'] [VpxaVMAP::Invoke] Command returned successfully
2011-07-11T00:09:59+02:00 host4 [2011-07-11 00:09:59.755 195A7B90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'

Splunk indexes each line as one event (as expected). there are only 3 esxi events here though, starting with syslog-ng_timestamp host4 [2011-07-11] 00:09:59....

any ideas on how to take esxi's timestamp as separators withouth changing the syslog-ng config (if possible at all) or using splunk forwarder etc.?

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...