topic Re: Props.conf and time_format, what am I missing in Getting Data In

Props.conf and time_format, what am I missing

howyagoin — Mon, 28 Sep 2020 09:50:23 GMT

I'm missing something, not sure what...I've got some GMT timestamped logs that Splunk didn't magically guess correctly for timezone/date format, so I figured I'd give it a helping hand in props.conf.

My events look like this:

20110825 000702531+0000 hostname01 daemon 13020 1594146607 1594146607 Note;FoobleWibble(50/6) user=name@domain.com:cmd=run :fromhost=1.2.3.4

And I've added the following into props.conf for the directory where the files are located:

[source::/import/logs/files]
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_FORMAT = %Y%m%d %H%M%S%3N%z

GNU date seems to interpret that correctly, and from the reading I've seen on strptime on linux, this should be right, but it just doesn't seem to get it.

Do I really need a transforms.conf entry for this? Or time_prefix? Since the timestamp is at the start of the line, I was thinking the time_format would be enough.

Am I missing something obvious?

Re: Props.conf and time_format, what am I missing

hexx — Mon, 28 Sep 2020 09:50:50 GMT

I would assume that MAX_TIMESTAMP_LOOKAHEAD = 22 should be enough. How is Splunk getting the date wrong, exactly? Could you show us an example?

Re: Props.conf and time_format, what am I missing

howyagoin — Thu, 08 Sep 2011 04:00:34 GMT

Yeah:

1   9/8/11 1:35:33.000 PM   20110907 185959292+0000 blah blah 22415 691265462 691265462 Note;blah2
2   9/8/11 1:35:33.000 PM   20110907 185957218+0000 blah blah 22415 691265200 691265200 Note;blah123
3   9/8/11 1:35:33.000 PM   20110907 185956595+0000 blah blah 22415 691265275 691265275 Note;blah123123

From a timechart point of view, everything is timestamped at the moment of reading the file.

Log errors:
AggregatorMiningProcessor - Too many events (400K) with the same timestamp: incrementing timestamps 4 seconds into the future to insure retrievability

Re: Props.conf and time_format, what am I missing

woodcock — Fri, 29 May 2015 13:23:27 GMT

First of all, I would not use source if at all possible; I have always used sourcetype for this. Secondly, your timestamps are 22 characters long, not 25. Try this:

[MySourceType]
MAX_TIMESTAMP_LOOKAHEAD = 22
TIME_FORMAT = %Y%m%d %H%M%S%3N%z

Also, you need to make sure that this is deployed to your indexers, which is where timestamping occurs, not to your forwarder.