Timestamp set to Midnight

[romeoszakal]

The timestamp of an application log file is always being set to midnight, an example line in the logs is:

02/05/17 14:47:21 IN:[(1)29RET_YSCO      (2)(28)92 RINQ(28)1005183824(28)(28)10(28)990364911(28)457(28)7.50(28)Y(3)0CF9(4)]

This is given an event time of 02/05/2017 00:00:00.000

The sourcetype config for this log in props.conf is as follows:

[flat_file]
SHOULD_LINEMERGE = false
LINE_BREAKER = [\r\n]+
MAX_EVENTS = 1
TIME_FORMAT = %d/%m/%y %H:%M:%S
MAX_DAYS_AGO = 1
MAX_TIMESTAMP_LOOKAHEAD = 20

When I test the sourcetype config with the same log file using the Add Data / Set Source Type functioanlity in Splunk it assigns the time from the log file correctly!

I'm ingesting numerous other logs (of different format) from the same host without issue.

Can anyone see why the event time is always set to midnight for this log?

[DalJeanis]

I have too, so it's not that much of a reach. That's why I used \s+, which will take away pretty much any white space characters.

[romeoszakal]

Tried adding both the \t and \s+ suggestion (and restarted the server) but no change!

Really baffled by this one! even if it is ignoring the TIME_FORMAT in the props.conf, I don't understand why it is always assigning the time of midnight!

[dineshraj9]

Can you try this -

[flat_file]
TIME_PREFIX = ^(?=\s*)
MAX_TIMESTAMP_LOOKAHEAD = 22
TIME_FORMAT = %d/%m/%y %T
LINE_BREAKER = ([\n\r]+)
SHOULD_LINEMERGE = False

[romeoszakal]

Tried, no difference unfortunately, thanks for your suggestions though.

To see if the props.conf was being read, I then removed the LINE_BREAKER and SHOULD_LINEMERGE entry, this had the expected affect of merging lines, so the props.conf entry is being used.

I then removed all the TIME entries, so Splunk will go back to AUTO, no difference. The timestartpos is always 0 and timeendpos is always 8?? Don't understabnd why it thinks this is the case!

04/05/17 20:58:27 OUT:[(1)22RET_YSCO (2)(28)38 VSNG(28)OK(28)69000359(28)10523189(28)H(28)NGUYEN(28)(3)(4)]

time - 2017-05-04T00:00:00.000+10:00

date_mday - 4

date_month - may

date_wday - thursday

date_year - 2017

date_zone - local

timeendpos - 8

timestartpos - 0

Default
host - gis-syco-01

index - crown

punct - //:::[()____()()()()()()()()()()]

source - /sysC/logs/simphony/MICpst.socket

sourcetype - flat_file

splunk_server - MIT-SPLUNK-T1

[romeoszakal]

As mentioned, if I use the Add Data functionality in Splunk, the sourcetype section shows the time is recognised correctly with the flat_file sourcetype, its even correct with the default!

[adonio]

does it extracts the correct timestamp using the Add Data function in GUI?
made couple of attempts and indeed (in my splunk 6.5.2 on windows 10) it is extracted correctly with default configurations. maybe you have another props with the same sourcetype that overrides these configurations?

[romeoszakal]

Yes with the Add data function in GUI the time is extracted correctly with default and the sourcetype of flat_file!

[adonio]

hello,
can you try and add TIME_PERFIX = ^ to your props.conf?

[romeoszakal]

I've added TIME_PREFIX = ^ to the props.conf but unfortunately no change.