topic Re: Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve? in Getting Data In

Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve?

johnny21 — Thu, 23 May 2019 12:00:44 GMT

Splunk noobie here: When I configure my props.conf file like the first snippet I get multiple events showing as a single event.

[splunk@localhost ~]$ cat /opt/splunk/etc/apps/search/local/props.conf
 [sophos:syslog]
 TIME_PREFIX = <..>
 TIME_FORMAT = %Y:%m:%D-%H:%M:%S
 MAX_TIMESTAMP_LOOKAHEAD = 19

So in an attempt to resolve I modify props.conf to look like this:

[sophos:syslog]
TIME_PREFIX = <..>
TIME_FORMAT = %Y:%m:%D-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = false

I get the event issue resolved BUT then my dates appear to go backward at the beginning of the new day...

Any ideas/thoughts on how to resolve?

Re: Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve?

richgalloway — Thu, 23 May 2019 12:36:23 GMT

You asked a similar question (https://answers.splunk.com/answers/743872/splunk-date-going-backwards.html) and accepted an answer for it. How is this different?
I see 4 events in your screen shot. All of them have the same timestamp. How is that "backwards"?
Remember, by default, Splunk displays the newest events first.

Re: Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve?

johnny21 — Thu, 23 May 2019 13:17:13 GMT

Hey Rich - that is correct, I didn't want to conflate the issue as "technically" the props.conf as answered in that question addresses that one issue...it just also introduces a new one. I will try to clarify:

I am attaching a screenshot of what I mean by going backwards in the answers section below (as I can't attach screenshots in replies).

The blue line represents 11:59:59 then the red line takes us back to a previous date and inserts events from 12:00:00 and on to a previous day. The newest events are not being shown first; the newest event is at the end of the second red arrow.

Re: Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve?

johnny21 — Thu, 23 May 2019 13:55:45 GMT

Additional information for context on what I mean by "going backwards".

Re: Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve?

marycordova — Thu, 23 May 2019 18:11:56 GMT

You might have a typo in your TIME_FORMAT variables, specifically "D" instead of "d", you should not need line merge for these events.

[sophos:syslog]
TIME_PREFIX = <\d\d>
TIME_FORMAT = %Y:%m:%d-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19

Also, the "backwards" log events might be related to timezones. What timezone are your logs in and what timezone is your Splunk UI configured to display?

You might want to add a TZ = <timezone identifier> to props.conf also: https://answers.splunk.com/answers/127193/where-are-splunk-valid-tz-options-in-propsconf.html

Re: Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve?

johnny21 — Wed, 30 Sep 2020 00:42:32 GMT

Thanks for the advice Mary!

So far things are looking good, I'll confirm everything works tomorrow morning once we hit a new day.

Current props.conf
[sophos:syslog]
TIME_PREFIX = <\d\d>
TIME_FORMAT = %Y:%m:%d-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TZ = US/Eastern

Re: Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve?

johnny21 — Fri, 24 May 2019 11:22:21 GMT

Seems to have resolved my issue, many thanks!