Why are we Receiving Incorrect sourcetypes for ESXi syslog?

[justynap_ldz]

Hello Splunkers,

We configured Splunk Add-on for VMware ESXi Logs on one of our Heavy Forwarders as in: https://docs.splunk.com/Documentation/AddOns/released/VMWesxilogs/Install

However, we can see a huge bunch of wrongly extracted sourcetypes:
e.g.:
vmware:esxlog:--
vmware:esxlog:ERROR
vmware:esxlog:INFO
vmware:esxlog:NoneZ
vmware:esxlog:WARNING
vmware:esxlog:a
vmware:esxlog:a-cli-info-python
vmware:esxlog:a-dabc
vmware:esxlog:a-e
vmware:esxlog:a-vsan-task-tracker
vmware:esxlog:ab
vmware:esxlog:abc
vmware:esxlog:af
vmware:esxlog:althSystemImpl

We tried to add additional regex in set_syslog_sourcetype in transforms.conf, but then the events stopped coming in at all.

Our config files are as follows (all on Heavy Forwarder):
inputs.conf

[monitor:///opt/splunk/var/log/remote/syslog-tlxfr*.log]
disabled = false
index = vmware-esxilog
sourcetype = vmw-syslog

props.conf

[vmw-syslog]
TRANSFORMS-vmsysloghost=set_host
TRANSFORMS-vmsyslogsourcetype = set_syslog_sourcetype
MAX_TIMESTAMP_LOOKAHEAD = 20

transforms.conf

[set_host]
REGEX = ^(?:\w{3}\s+\d+\s+[\d\:]{8}\s+([^ ]+)\s+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[set_syslog_sourcetype]
REGEX = ^(?:(?:\w{3}\s+\d+\s+[\d\:]{8})|(?:<\d+>)?(?:(?:(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(?:Z|[\+\-][\d\:]{3,5})?))|(?:NoneZ)?)|(?:\w{3}\s+\w{3}\s+\d+\s+[\d\:]{8}\s+\d{4}))\s[^ ]+\s+([A-Za-z\-]+)(?:[^:]*)[:\[]
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::vmware:esxlog:$1

[esx_hostd_fields_6x]
REGEX = ^(?:(?:\w{3}\s+\d+\s+[\d\:]{8})|(?:<(\d+)>)?(?:(?:(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(Z|[\+\-][\d\:]{3,5})?))|(?:NoneZ)?)|(?:\w{3}\s+\w{3}\s+\d+\s+[\d\:]{8}\s+\d{4}))\s[^ ]+\s+([^\[\:]+):\s(?:(?:[\d\-:TZ.]+)\s*)?(\w+)\s*(?:\S+\[\S+\])?\s*\[(?:[^\s\]]+)\s*(?:sub=([^\s\]]+))?\s*(?:opID=([^\s\]]+))?(?:[^]]+?)?\]\s*(.*)$
FORMAT = Pri::$1 Offset::$2 Application::$3 Level::$4 Object::$5 opID::$6 Message::$7

Does anyone have any idea how to solve it? Seems to be simple, but we are stuck
Greetings,
Justyna

[PickleRick]

The add-on docs say that the supported method of ingesting data is to send them over the network https://docs.splunk.com/Documentation/AddOns/released/VMWesxilogs/Setup

You're pulling the logs directly from the files.

Therefore there is a bit different structure of the raw events - syslog events sent over the wire more or less adhere to the typical syslog message structure, and the set_syslog_sourcetype transform relies on this structure to extract and recast sourcetype from the message. Most probably the file stored on the ESXi server does not have the syslog header and the transform hits wrong fields and parses wrong part of the event as sourcetype.

You should either reconfigure the ingestion method to sending the events over the network or set up separate inputs for separate sourcefiles with specific sourcetypes (I suppose separate files correlate more or less with separate source processes) and completely disable the sourcetype-recasting transform.

[justynap_ldz]

Hi @PickleRick,

thanks for replying!

Data comes in over tcp:514 (port and protocol preferred by VMWare team), saved to disk on Splunk Heavy Forwarder which is our syslog server not only for esxi syslog, but also ontap syslog . There are already around 300 esxi hosts sending data that way.  We were thinking that data is pushed to Splunk from ESXi hosts, not the other way round...
With such a config, isn't syslog data sent over the network? I am bit confused now..

[PickleRick]

Ahh, my mistake. I misinterpreted the file path in inputs.conf. I thought you were mounting remote storage from ESXi by some means (NFS? sshfs?) and pulling log files from there.

So you're using some kind of syslog daemon on the HF machine and then the forwarder pulls data from the logs that syslog daemons writes to?

Most probably your syslog solution strips some headers or reorganizes the data in some way before writing it to disk.

There are many possible approaches to this situation

1. Correct the transform to catch proper field (might not be compatible with future add-on upgrades and you might have to adjust the resulting transform by hand after such upgrades)

2. Correct the syslog daemon configuration so it writes the events literarily as it was received from network to the file - relatively foolproof solution.

3. Get rid of the intermediate logfile layer - reorganize your syslog receiving process to receive events from network and send them directly to HEC input on your HFs/indexers by means of sc4s or rsyslog - most flexible solution but requires some work.