topic Re: Timestamp set to Midnight in Splunk Search

Timestamp set to Midnight

romeoszakal — Tue, 29 Sep 2020 13:54:34 GMT

The timestamp of an application log file is always being set to midnight, an example line in the logs is:

02/05/17 14:47:21 IN:[(1)29RET_YSCO      (2)(28)92 RINQ(28)1005183824(28)(28)10(28)990364911(28)457(28)7.50(28)Y(3)0CF9(4)]

This is given an event time of 02/05/2017 00:00:00.000

The sourcetype config for this log in props.conf is as follows:

[flat_file]
SHOULD_LINEMERGE = false
LINE_BREAKER = [\r\n]+
MAX_EVENTS = 1
TIME_FORMAT = %d/%m/%y %H:%M:%S
MAX_DAYS_AGO = 1
MAX_TIMESTAMP_LOOKAHEAD = 20

When I test the sourcetype config with the same log file using the Add Data / Set Source Type functioanlity in Splunk it assigns the time from the log file correctly!

I'm ingesting numerous other logs (of different format) from the same host without issue.

Can anyone see why the event time is always set to midnight for this log?

Re: Timestamp set to Midnight

adonio — Tue, 02 May 2017 13:55:59 GMT

hello,
can you try and add TIME_PERFIX = ^ to your props.conf?

Re: Timestamp set to Midnight

DalJeanis — Tue, 29 Sep 2020 13:54:55 GMT

Your timestamp is only 18 long including the space at the front (so it's < 20). There is exactly one space between data and time in the data and the format, so the basics are covered.

1) Try adonio's suggestion first. TIME_PREFIX = ^

If that doesn't fix the issue, then here's the next two shots in the increasingly pitch black dark...

2) Try setting MAX_DAYS_AGO to 2 just in case there's a UTC vs local time thing going on.

3) Any chance it's somehow not being handled as [flat_file]?

Re: Timestamp set to Midnight

romeoszakal — Wed, 03 May 2017 00:22:00 GMT

I've added TIME_PREFIX = ^ to the props.conf but unfortunately no change.

Re: Timestamp set to Midnight

romeoszakal — Tue, 29 Sep 2020 13:55:11 GMT

1) Added TIME_PREFIX = ^
2) Changed MAX_DAYS_AGO to 2
3) The sourcetype is being recorded correctly against the event.

Unfortunatley the time is still always assigned as midnight!

Example:
03/05/17 11:01:12 Msg: OUT:[(1)65RET_YSCO (2)(28)15 VSNG(28)OK(28)68975261(28)12689300(28)D(28)KALRA(28)(3)(4)]

Time

time - 2017-05-03T00:00:00.000+10:00

date_mday - 3

date_month - may

date_wday - wednesday

date_year - 2017

date_zone - 600
timeendpos - 8

timestartpos - 0

Default
host - gis-syco-01

index - crown

punct - //::::[()_____()()()()()()()()()()]

source - /sysC/logs/simphony/MICpst.debug

sourcetype - flat_file

splunk_server - MIT-SPLUNK-T1

Re: Timestamp set to Midnight

DalJeanis — Wed, 03 May 2017 22:56:19 GMT

Okay, one more shot in the dark. Suppose that innocent looking space is actually some other character? Try this one.

TIME_FORMAT = %d/%m/%y\s+%H:%M:%S

By the way, I assume you either made the conf changes through splunkweb and the CLI, or restarted splunkd to pick up the new conf file if you made the changes manually?

https://docs.splunk.com/Documentation/Splunk/6.5.3/Admin/Configurationfilechangesthatrequirerestart

The next shot in the next dark is all the way over the horizon and into the next ocean over... take your timestamp and verify exactly what characters in unicode are in the slashes and colons...

Hmm. Before I did that, I'd feed in a file with a 5/1 or prior date and see if it's getting 5/1 or 5/2 for that. I'm expecting it would have to get the current date, if it's not properly identifying the timestamp as a timestamp. If I'm wrong there, then we start beating the thickets for how it got the right date, and work from there.

Re: Timestamp set to Midnight

adonio — Thu, 04 May 2017 01:29:45 GMT

just adding a thought that maybe (huge maybe but I have seen it happen before) this space is a tab
in that case:

TIME_FORMAT = %d/%m/%y\t%H:%M:%S

Re: Timestamp set to Midnight

DalJeanis — Thu, 04 May 2017 02:27:25 GMT

I have too, so it's not that much of a reach. That's why I used \s+, which will take away pretty much any white space characters.

Re: Timestamp set to Midnight

romeoszakal — Thu, 04 May 2017 07:38:54 GMT

Tried adding both the \t and \s+ suggestion (and restarted the server) but no change!

Really baffled by this one! even if it is ignoring the TIME_FORMAT in the props.conf, I don't understand why it is always assigning the time of midnight!

Re: Timestamp set to Midnight

dineshraj9 — Thu, 04 May 2017 09:14:26 GMT

Can you try this -

[flat_file]
TIME_PREFIX = ^(?=\s*)
MAX_TIMESTAMP_LOOKAHEAD = 22
TIME_FORMAT = %d/%m/%y %T
LINE_BREAKER = ([\n\r]+)
SHOULD_LINEMERGE = False

Re: Timestamp set to Midnight

romeoszakal — Tue, 29 Sep 2020 13:56:58 GMT

Tried, no difference unfortunately, thanks for your suggestions though.

To see if the props.conf was being read, I then removed the LINE_BREAKER and SHOULD_LINEMERGE entry, this had the expected affect of merging lines, so the props.conf entry is being used.

I then removed all the TIME entries, so Splunk will go back to AUTO, no difference. The timestartpos is always 0 and timeendpos is always 8?? Don't understabnd why it thinks this is the case!

04/05/17 20:58:27 OUT:[(1)22RET_YSCO (2)(28)38 VSNG(28)OK(28)69000359(28)10523189(28)H(28)NGUYEN(28)(3)(4)]

time - 2017-05-04T00:00:00.000+10:00

date_mday - 4

date_month - may

date_wday - thursday

date_year - 2017

date_zone - local

timeendpos - 8

timestartpos - 0

Default
host - gis-syco-01

index - crown

punct - //:::[()____()()()()()()()()()()]

source - /sysC/logs/simphony/MICpst.socket

sourcetype - flat_file

splunk_server - MIT-SPLUNK-T1

Re: Timestamp set to Midnight

romeoszakal — Thu, 04 May 2017 11:18:19 GMT

As mentioned, if I use the Add Data functionality in Splunk, the sourcetype section shows the time is recognised correctly with the flat_file sourcetype, its even correct with the default!

Re: Timestamp set to Midnight

adonio — Thu, 04 May 2017 15:10:10 GMT

does it extracts the correct timestamp using the Add Data function in GUI?
made couple of attempts and indeed (in my splunk 6.5.2 on windows 10) it is extracted correctly with default configurations. maybe you have another props with the same sourcetype that overrides these configurations?

Re: Timestamp set to Midnight

alemarzu — Thu, 04 May 2017 15:27:13 GMT

Hi romeo, any internal error regarding bad recognition timestamp or something similar ?

Re: Timestamp set to Midnight

romeoszakal — Fri, 05 May 2017 00:05:14 GMT

Yes with the Add data function in GUI the time is extracted correctly with default and the sourcetype of flat_file!

Re: Timestamp set to Midnight

DalJeanis — Fri, 05 May 2017 00:08:53 GMT

@somesoni2, @woodcock, @jkat54, any ideas on this one?

Re: Timestamp set to Midnight

cpetterborg — Fri, 05 May 2017 00:55:25 GMT

What happens if you don't specify the TIME_FORMAT?

Re: Timestamp set to Midnight

romeoszakal — Fri, 05 May 2017 01:11:46 GMT

No internal errors for these logs!

Re: Timestamp set to Midnight

romeoszakal — Fri, 05 May 2017 06:14:11 GMT

I have tried this, no difference!

Re: Timestamp set to Midnight

alemarzu — Fri, 05 May 2017 13:00:53 GMT

Is this happening on a standalone server ?