topic Re: Why isn't the timestamp being recognized ? in Getting Data In

Why isn't the timestamp being recognized ?

ddrillic — Sat, 30 Sep 2017 21:14:19 GMT

We have a case that looks like this -

So, the events are not broken by the timestamp. Is it because of the underscore between the date and time?

Re: Why isn't the timestamp being recognized ?

skoelpin — Sat, 30 Sep 2017 21:30:24 GMT

Most likely the underscores are throwing it off. What's your props.conf look like?

Re: Why isn't the timestamp being recognized ?

gjanders — Mon, 02 Oct 2017 12:53:33 GMT

Splunk has a number of built in time formats, I would recommend you manually parse your timestamps to improve indexing performance and ensure that you have the timestamps consistently recognised.

In this case in your props.conf

[replace_with_your_sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d_%H:%M:%S.%3N

Also refer to configure timestamp recognition. I agree with skoelpin it's probably the _ throwing the default timestamp parsing off.

Re: Why isn't the timestamp being recognized ?

ddrillic — Mon, 02 Oct 2017 13:29:10 GMT

Gorgeous - thank you @garethatiagg

Re: Why isn't the timestamp being recognized ?

skoelpin — Mon, 02 Oct 2017 13:34:19 GMT

Remember that this is a start and you should apply line breaking to your props.conf aswell.. This will take some load off your indexers

Re: Why isn't the timestamp being recognized ?

gjanders — Tue, 29 Sep 2020 16:00:10 GMT

The documentation also covers line breaking

This is a slightly more advanced topic but potentially worth discussing here, the default for the MAX_EVENTS in props.conf is 256 lines at the time of writing, if you always have single line events you could switch SHOULD_LINEMERGE=false on and never line merge (this entry would go in the props.conf)

Alternatively if you have very large events (>1000 lines) you could make a more complicated LINE_BREAKER setting and change SHOULD_LINEMERGE to false such as:

[replace_with_your_sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d_%H:%M:%S.%3N
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}_\d

Of course the above would require testing, and it would only make sense if you have larger multi-line events (I've used examples like the above for events >1000 lines), doing this can remove pressure from the aggregation queue as the merging will move to the parsing queue...

Re: Why isn't the timestamp being recognized ?

ddrillic — Wed, 04 Oct 2017 17:13:37 GMT

Btw, this by itself did it ; -) with no line breaking