topic TIME_FORMAT in props.conf seems to not be working in Getting Data In

TIME_FORMAT in props.conf seems to not be working

brent_weaver — Thu, 26 Nov 2020 17:05:05 GMT

Hello all... I have events that have a timestamp that starts with:

2014-05-07 13:12:27 2910 ...

The trailing # (2910) is variable in length and this seems to mess things up. I have been setting this in my props.conf for this sorucetype:

TIME_FORMAT = %Y-%m-%d %H:%M:%S

This is not working. What am I missing?

Re: TIME_FORMAT in props.conf seems to not be working

bowesmana — Wed, 25 Nov 2020 22:31:32 GMT

Did you mean

TIME_FORMAT=%Y-%m-%d...

i.e. d not m?

Re: TIME_FORMAT in props.conf seems to not be working

richgalloway — Wed, 25 Nov 2020 22:34:18 GMT

The TIME_FORMAT setting should be "%Y-%m-%d %H:%M:%S", but that probably was a typo in the question rather than the code.

Please elaborate at some length on "not working". What is the expected result? What is the actual result? Are there any error messages in splunkd.log? If so, what do they say?

Re: TIME_FORMAT in props.conf seems to not be working

inventsekar — Thu, 26 Nov 2020 00:16:42 GMT

The trailing # (2910) is variable in length and this seems to mess things up///

whats that 2910?

is it the sub-seconds, etc.?!?!

do you want to keep this with the timestamp or not needed?

hope you corrected your typo (as @bowesmana mentioned)

Re: TIME_FORMAT in props.conf seems to not be working

brent_weaver — Thu, 26 Nov 2020 17:06:20 GMT

Hey sorry about the type, its fixed. No that trailing # is not sub seconds, its a random integer. I do not want to include this in the format.

Thanks for taking the time to respond and help me out!

Re: TIME_FORMAT in props.conf seems to not be working

brent_weaver — Thu, 26 Nov 2020 17:12:28 GMT

Not working as in the timestamps get really crazy. Where the year is 2014 in the raw event, in Splunk it shows as current time... This clearly indicates that it is not in fact being read. I am certain it is a config issue that I am overlooking.

My props looks like:

[testst] DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TIME_FORMAT = %Y-%m-%d %H:%M:%S

I feel like there is just some param I am overlooking. Thank you for taking the time to help me out here.

Splunkers rock!

Re: TIME_FORMAT in props.conf seems to not be working

richgalloway — Thu, 26 Nov 2020 19:15:18 GMT

I suspect Splunk is reluctant to index something that might be 6 years old. There should be a message to that effect in splunkd.log.

Try adding

MAX_DAYS_AGO = 3650

to your props.conf. That will tell Splunk to accept dates up to 10 years old.

Re: TIME_FORMAT in props.conf seems to not be working

brent_weaver — Thu, 26 Nov 2020 20:01:41 GMT

Yes I did.

Re: TIME_FORMAT in props.conf seems to not be working

inventsekar — Thu, 26 Nov 2020 20:43:32 GMT

Hi @brent_weaver Any ideas how this issue was fixed(for future readers reference and for our learnings as well).. maybe, accept Rich's or any post closer to your fix, choose it and accept it as the solution.

Best Regards,

Sekar

PS - your karma points will be my 2 cents!

Re: TIME_FORMAT in props.conf seems to not be working

brent_weaver — Fri, 27 Nov 2020 18:24:43 GMT

Hey there... Solid catch, that worked. Yes the timestamp was outside of the range and therefore problematic. Thank you very much for helping me out here, and on thanksgiving nonetheless. There is nothing like the Splunk community!