topic Re: Miliseconds in timestamp are not extracted in Getting Data In

Miliseconds in timestamp are not extracted

mvagionakis — Wed, 30 Sep 2020 01:39:57 GMT

Hello all,

I have again something strange with my logs, the milliseconds in the _time field are not detected despite the applied props.conf parameters.

Here how it looks every line of my log:

1234 08/08/2019 15:08:56:924 123456789 0000049T6 TOTOPROCESS INF TOTO settings - process timeout set to 70 s

Here my props.conf:

 [mysourcetype] 
 TIME_PREFIX =^\d+\s\d{1,2}\/\d{1,2}\/\d{4}\s\d{1,2}\:\d{1,2}\:\d{1,2}\:\d{1,4}\s
 TIME_FORMAT = %d/%m/%Y %H:%M:%S:%3N
 MAX_TIMESTAMP_LOOKAHEAD = 29 
 TZ = GMT
 BREAK_ONLY_BEFORE =^\d+\s\d{1,2}\/\d{1,2}\/\d{4}\s\d{1,2}\:\d{1,2}\:\d{1,2}\:\d{1,4}\s

I tried all the possible solutions that I could find in the forum but nothing works.

The timestamp shows always three zeros for the milliseconds.

8/8/19 3:08:56.000 PM

I tried also by disabling the time_prefix, changing the time_format parameters,etc, but nothing helps.
At the beginning I thought that the props.conf were not being applied but I changed the "TZ" parameter (for testing purposes) and it was immediately applied so I don't think that the UF ignores my configuration.

I don't have any solution for the moment and any suggestion is welcome.

Thank you in advance.
Michael

Re: Miliseconds in timestamp are not extracted

neha898 — Thu, 08 Aug 2019 14:16:05 GMT

Hi There,
Please use TIME_FORMAT = %d/%m/%Y %H:%M:%S:%f
This works for me every time.

Re: Miliseconds in timestamp are not extracted

gcusello — Thu, 08 Aug 2019 14:19:43 GMT

Hi mvagionakis,
I'm not sure if there are some spaces at the beginning of each row, anyway try this:
(without spaces)

TIME_PREFIX =^\d+\s
TIME_FORMAT = %d/%m/%Y %H:%M:%S%:%3N
MAX_TIMESTAMP_LOOKAHEAD = 34

(with spacese):

TIME_PREFIX =^\s+\d+\s
TIME_FORMAT = %d/%m/%Y %H:%M:%S%:%3N
MAX_TIMESTAMP_LOOKAHEAD = 34

Bye.
Giuseppe

Re: Miliseconds in timestamp are not extracted

mayurr98 — Thu, 08 Aug 2019 14:35:49 GMT

okay I misunderstood your question.
here you go,

  [mysourcetype] 
  TIME_PREFIX = \d+\s+
  TIME_FORMAT = %d/%m/%Y %H:%M:%S%:%3N
  MAX_TIMESTAMP_LOOKAHEAD = 25

Re: Miliseconds in timestamp are not extracted

mvagionakis — Thu, 08 Aug 2019 16:13:10 GMT

hey neha898,

it doesn't work 😞

Re: Miliseconds in timestamp are not extracted

mvagionakis — Thu, 08 Aug 2019 16:13:30 GMT

hey mayurr98

it's not better 😞

Re: Miliseconds in timestamp are not extracted

mvagionakis — Thu, 08 Aug 2019 16:14:50 GMT

Hey gcusello ,

thanks for your suggestion but I've already tried it but without success.

ps: no spaces at the beginning of the line.

Re: Miliseconds in timestamp are not extracted

neha898 — Wed, 30 Sep 2020 01:42:02 GMT

Hi mvagionakis,
This should solve your issue:
SHOULD_LINEMERGE=false
TIME_FORMAT=%d/%m/%Y %H:%M:%S:%f
TIME_PREFIX=^\d+

Your TIME_PREFIX is incorrect

Re: Miliseconds in timestamp are not extracted

gcusello — Fri, 09 Aug 2019 06:02:52 GMT

Hi mvagionakis,
where did you put this props.conf?
It must be located on Indexers.

Bye.
Giuseppe

Re: Miliseconds in timestamp are not extracted

mvagionakis — Fri, 09 Aug 2019 09:13:22 GMT

hello neha898,

once your config applied, UF stopped forwarding so I rollback to my old config.

Re: Miliseconds in timestamp are not extracted

neha898 — Fri, 09 Aug 2019 14:02:36 GMT

These configs need to be applied on Indexer, not on UFs

Re: Miliseconds in timestamp are not extracted

mvagionakis — Fri, 09 Aug 2019 14:08:03 GMT

it is applied also on indexers.

Re: Miliseconds in timestamp are not extracted

gcusello — Fri, 09 Aug 2019 14:11:37 GMT

If you have heavy Forwarders, it must be appliaed on HFs.
Bye.
Giuseppe

Re: Miliseconds in timestamp are not extracted

mvagionakis — Fri, 09 Aug 2019 14:17:05 GMT

I applied it on indexers also....but still the same...

something strange happened....I disabled completely the props.conf and the timestamp is finally recognized by splunk without problem....

I also asked from the developer to remove the first digits so to start every line with the timestamp....I hope that it will be better once the modification done.

thank you again.