vmware esxlog and datetime parsing

tfechner — Wed, 30 Sep 2020 04:43:57 GMT

Hi,

I have trouble to parse the timestamp of ESX-logs.

The esx-syslog:
Mar 18 21:15:02 hostname 2020-03-18T20:15:02.109Z hostname hostd-probe: info hostd-probe[FFA22350]
and antoher log:
Mar 18 21:15:02 hostname 2020-03-18T20:15:02Z hostname hostd-probe: info hostd-probe[FFA22350]

Because of some special multiline log I cut with SEDCMD the trailing splunk date "Mar 18 21:15:02". But this is done at the end of the parsing phase during indexing. So splunk tries first to read the date from the whole log. There are two formats, I like to have the date with milliseconds.

Problems: the timezone is not recognized! I have an offset from 1 or 2 hours. And the milliseconds are not extracted.

I tried:
- TZ = UTC
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N
-> does not work (maybe the T in the format string is not valid)
- added some lines in datetime.xml and referenced it with DATETIME_CONFIG in props.conf (checked with btool )

    <define name="_time_without_zone" extract="hour, minute, second, subsecond">
      <text><![CDATA[(?<=T)]]></text>
            <use name="_hour"/>
      <text><![CDATA[:]]></text>
            <use name="_minute"/>
      <text><![CDATA[:]]></text>
            <use name="_second"/>
      <text><![CDATA[(?:(?: \d{4})?[:,\.](\d+))? {0,2}]]></text>
    </define>
<timePatterns>
      <use name="_time_without_zone"/>
      <use name="_time"/>
      <use name="_time_without_subsec"/>
      <use name="_time_no_sub"/>
      <use name="_time_esxi_4x"/>
      <!-- Uncomment the below comments if ESX 4 exists in the environment
      <use name="_time_esx_4x"/> -->
</timePatterns>

SO how can I extract the correct date and timezone?

Torsten

Re: vmware esxlog and datetime parsing

to4kawa — Sat, 28 Mar 2020 05:39:40 GMT

[ esx ]
CHARSET=UTF-8
SHOULD_LINEMERGE=false
disabled=false
TIME_FORMAT=%FT%T.%3QZ
TIME_PREFIX=\w{4,}\s
LINE_BREAKER=([\r\n]+)

TIME_PREFIX is hostname ,your real hostname is with [^A-z0-9_]+, change REGEX.

Re: vmware esxlog and datetime parsing

tfechner — Tue, 31 Mar 2020 09:34:19 GMT

works now - thank you.
does splunk handle %F and %T different than %H:%M....
your props seems to be very similar to mine.

topic Re: vmware esxlog and datetime parsing in Getting Data In

vmware esxlog and datetime parsing

Re: vmware esxlog and datetime parsing

Re: vmware esxlog and datetime parsing