basically this is not a real question but more an analysis of the somewhat broken syslog format of some messages issued by ESXi. No answers are expected, but comments are welcome, especially if you are hit by the problems described here...
vmw-syslog, which is used by the installed TA_esxilogs as an temporary sourcetype. If you use this TA, then searching and finding this sourcetype in your index may be an indication that you are affected by the problems described in this article.
Time for examining ESXi syslogs on the network packet level with Wireshark. We captured the syslog traffic on the (r)syslog-server at the incoming network interface to catch the packets in exactly the same format as they are sent from the ESXi host.
The format of some ESXi syslog messages is badly broken!
ESXi uses a funny kind of multiline syslog message for a few events. And these events are chunked into packets of less than 1024 bytes regardless of the syslog packte size set on the ESXi host (see above).
The first packet (a "syslog line") is correct according to the syslog packet format:
So far, so good. BUT the next 2-3 continuation lines are just totally mangled up and wrong for syslog packets (according to the RFC, which by the way does not define multiline syslogs...):
This funny kind of continuation lines goes on, until the current "long event message text" of this single message is processed completly. Then the next "normal" syslog message follows. This broken format cannot be fixed easily with props or transforms inside Splunk!
To repeat very clearly: This is the format ORIGINATING from the ESXi host as captured directly on the wire without any processing!
Attached is a sample of one of the broken messages captured by Wireshark: textual output from the Wireshark capture. IP- and MAC-addresses have been shortened.
We will file a bug report with VMware, but expectations regarding getting a fix is very low...
Hope, this helps others wondering about issues with ESXi logging...
Have fun and happy Splunking!