Getting Data In

Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve?

johnny21
Path Finder

Splunk noobie here: When I configure my props.conf file like the first snippet I get multiple events showing as a single event.

[splunk@localhost ~]$ cat /opt/splunk/etc/apps/search/local/props.conf
 [sophos:syslog]
 TIME_PREFIX = <..>
 TIME_FORMAT = %Y:%m:%D-%H:%M:%S
 MAX_TIMESTAMP_LOOKAHEAD = 19

alt text

So in an attempt to resolve I modify props.conf to look like this:

[sophos:syslog]
TIME_PREFIX = <..>
TIME_FORMAT = %Y:%m:%D-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = false

I get the event issue resolved BUT then my dates appear to go backward at the beginning of the new day...

alt text

Any ideas/thoughts on how to resolve?

0 Karma
1 Solution

marycordova
SplunkTrust
SplunkTrust

You might have a typo in your TIME_FORMAT variables, specifically "D" instead of "d", you should not need line merge for these events.

[sophos:syslog]
TIME_PREFIX = <\d\d>
TIME_FORMAT = %Y:%m:%d-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19

Also, the "backwards" log events might be related to timezones. What timezone are your logs in and what timezone is your Splunk UI configured to display?

You might want to add a TZ = <timezone identifier> to props.conf also: https://answers.splunk.com/answers/127193/where-are-splunk-valid-tz-options-in-propsconf.html

@marycordova

View solution in original post

marycordova
SplunkTrust
SplunkTrust

You might have a typo in your TIME_FORMAT variables, specifically "D" instead of "d", you should not need line merge for these events.

[sophos:syslog]
TIME_PREFIX = <\d\d>
TIME_FORMAT = %Y:%m:%d-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19

Also, the "backwards" log events might be related to timezones. What timezone are your logs in and what timezone is your Splunk UI configured to display?

You might want to add a TZ = <timezone identifier> to props.conf also: https://answers.splunk.com/answers/127193/where-are-splunk-valid-tz-options-in-propsconf.html

@marycordova

johnny21
Path Finder

Thanks for the advice Mary!

So far things are looking good, I'll confirm everything works tomorrow morning once we hit a new day.

Current props.conf
[sophos:syslog]
TIME_PREFIX = <\d\d>
TIME_FORMAT = %Y:%m:%d-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TZ = US/Eastern

0 Karma

johnny21
Path Finder

Seems to have resolved my issue, many thanks!

johnny21
Path Finder

Additional information for context on what I mean by "going backwards".

alt text

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You asked a similar question (https://answers.splunk.com/answers/743872/splunk-date-going-backwards.html) and accepted an answer for it. How is this different?
I see 4 events in your screen shot. All of them have the same timestamp. How is that "backwards"?
Remember, by default, Splunk displays the newest events first.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

johnny21
Path Finder

Hey Rich - that is correct, I didn't want to conflate the issue as "technically" the props.conf as answered in that question addresses that one issue...it just also introduces a new one. I will try to clarify:

I am attaching a screenshot of what I mean by going backwards in the answers section below (as I can't attach screenshots in replies).

The blue line represents 11:59:59 then the red line takes us back to a previous date and inserts events from 12:00:00 and on to a previous day. The newest events are not being shown first; the newest event is at the end of the second red arrow.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...