How to configure props.conf and transforms.conf to process the timestamps in my data?

[smudge797]

Have data that Splunk is struggling with and needs props.conf and transforms.conf.
The year/month/date followed by time hours minutes seconds are in the 3rd and 4th fields for each event:

20170129|4928|20170128|191928|14390803651|DIMPLE|DESAITT|V123456|EMPLOYEE|058000-5440 SOMEDATA|05800001 SOMEDATA5440|3681-1-01 SOMEDATA|Open|GLOBAL|058000|3681|SOMEDATA|MUMBAI|INDIA|

20170129|4928|20170129|191928|14390803651|DIMPLE|DESAITT|E123456|EMPLOYEE|058000-5440 SOMEDATA|05800001 SOMEDATA5440|3681-1-01 SOMEDATA|Open|GLOBAL|058000|3681|SOMEDATA|CONGO|AFRICA|

Any help to sort time stamps and the pipe separated fields would be much appreciated.

[s2_splunk]

You should be able to configure your sourcetype in props.conf like this:

TIME_PREFIX=^\d+\|\d+\|
TIME_FORMAT=%Y%m%d|%H%M%S

TIME_PREFIX basically says that your timestamp is following two sequences of digits between pipes at the beginning of each line.
TIME_FORMAT then describes the timestamp as year/month/date, followed by a pipe symbol, followed by hour/minute/second.
While you're at it, set MAX_TIMESTAMP_LOOKAHEAD to 25.

I haven't tested this, but it should be pretty straightforward. What timezone are these times in, can you be sure they all come in from the source in the same timezone?

[smudge797]

Looks good thanks! However Im seeing timestamps issue where in the events the leading zeros are trimmed off of the UTC_TIME field, so that midnight is represented as a single 0, 12:15 is represented as the 2 digit 15, etc

[s2_splunk]

Check out this answers post on how to configure datetime.xml to support multiple timestamp formats within the same log file.

If you have any influence over how those logs are written, you may want to encourage the developer to make your life a little easier. 😉