topic Re: Added a new entry into datetime.xml and I can't seem to get it to work in Getting Data In

Added a new entry into datetime.xml and I can't seem to get it to work

MedralaG — Tue, 29 Sep 2020 23:03:55 GMT

I need to add an additional time format for syslog data into the datetime.xml.
Following the docs I have created a copy of datetime.xml and copied it to /system/local,
added the following stanza to it:

define name="custom_date" extract="year, month, day">
<![CDATA[^<\d+>(\d{4})-(\d{2})-(\d{2})]]>
/define>
define name="custom_time" extract="hour, minute, second, zone">
<![CDATA[^<\d+>\d{4}-\d{2}-\d{2}T(\d{2}):(\d{2}):(\d{2}).\d{4}(\S\d{2}:\d{2})]]>
/define>

timePatterns>
use name="custom_time"/>

datePatterns>
use name="custom_date"/>

I have also created a props.conf in system/local with the following stanza:
[syslog]
DATETIME_CONFIG = /etc/system/local/datetime.xml

Below are example events that I am trying to get the extraction to work on:

-86-2019-01-25T18:26:30.4563+04:00 This is a test event
-86-2019-01-27T19:28:31.4563+00:00 This is a test event

When I try to ingest a file (via the gui) with those events in the file and select syslog as sourcetype I get a warning saying that it could not use strptime to parse the timestamp.
There is a highlight on the even that starts at the beginning of the line but ends at "+04:" , so for some reason it does not include the last two characters (00) in the timezone , even though the regex states that it should include it.

Could someone help out?

Re: Added a new entry into datetime.xml and I can't seem to get it to work

chrisyounger — Fri, 01 Feb 2019 18:40:12 GMT

make sure your TIME_PREFIX and TIME_STAMPLOOKAHEAD are both set correctly in props.conf

You custom_time should look like this:

^<\d+>\d{4}-\d{2}-\d{2}T(\d{2}):(\d{2}):(\d{2}).\d{4}(\S\+?\-?\d{2}:\d{2})

Re: Added a new entry into datetime.xml and I can't seem to get it to work

MedralaG — Tue, 29 Sep 2020 23:09:17 GMT

OK, I eventually got to the bottom of this.
The regex that I have put in there is fine , so is the remaining format of the xml file.
I initially discovered that if I create a new sourcetype (and not use "syslog") and get it to use the datetime.xml then it works, so it must have been something in the syslog props definition.
I eventually realised it was the MAX_TIMESTAMP_LOOKAHEAD which is set to 30 for syslog but for any new sourcetype (which will inherit the default settings) it will be set to 128.
The timestamp in the log example that I have ends on the 43rd character of the event.

Re: Added a new entry into datetime.xml and I can't seem to get it to work

chrisyounger — Mon, 04 Feb 2019 20:46:46 GMT

Nice work! Its extra confusing when the sourcetypes are changed during parsing or with a rename...