Getting Data In

Why does SHOULD_LINEMERGE setting appear to make the date go backwards and how can I resolve?

johnny21
Path Finder

Splunk noobie here: When I configure my props.conf file like the first snippet I get multiple events showing as a single event.

[splunk@localhost ~]$ cat /opt/splunk/etc/apps/search/local/props.conf
 [sophos:syslog]
 TIME_PREFIX = <..>
 TIME_FORMAT = %Y:%m:%D-%H:%M:%S
 MAX_TIMESTAMP_LOOKAHEAD = 19

alt text

So in an attempt to resolve I modify props.conf to look like this:

[sophos:syslog]
TIME_PREFIX = <..>
TIME_FORMAT = %Y:%m:%D-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = false

I get the event issue resolved BUT then my dates appear to go backward at the beginning of the new day...

alt text

Any ideas/thoughts on how to resolve?

0 Karma
1 Solution

marycordova
SplunkTrust
SplunkTrust

You might have a typo in your TIME_FORMAT variables, specifically "D" instead of "d", you should not need line merge for these events.

[sophos:syslog]
TIME_PREFIX = <\d\d>
TIME_FORMAT = %Y:%m:%d-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19

Also, the "backwards" log events might be related to timezones. What timezone are your logs in and what timezone is your Splunk UI configured to display?

You might want to add a TZ = <timezone identifier> to props.conf also: https://answers.splunk.com/answers/127193/where-are-splunk-valid-tz-options-in-propsconf.html

@marycordova

View solution in original post

marycordova
SplunkTrust
SplunkTrust

You might have a typo in your TIME_FORMAT variables, specifically "D" instead of "d", you should not need line merge for these events.

[sophos:syslog]
TIME_PREFIX = <\d\d>
TIME_FORMAT = %Y:%m:%d-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19

Also, the "backwards" log events might be related to timezones. What timezone are your logs in and what timezone is your Splunk UI configured to display?

You might want to add a TZ = <timezone identifier> to props.conf also: https://answers.splunk.com/answers/127193/where-are-splunk-valid-tz-options-in-propsconf.html

@marycordova

johnny21
Path Finder

Thanks for the advice Mary!

So far things are looking good, I'll confirm everything works tomorrow morning once we hit a new day.

Current props.conf
[sophos:syslog]
TIME_PREFIX = <\d\d>
TIME_FORMAT = %Y:%m:%d-%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TZ = US/Eastern

0 Karma

johnny21
Path Finder

Seems to have resolved my issue, many thanks!

johnny21
Path Finder

Additional information for context on what I mean by "going backwards".

alt text

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You asked a similar question (https://answers.splunk.com/answers/743872/splunk-date-going-backwards.html) and accepted an answer for it. How is this different?
I see 4 events in your screen shot. All of them have the same timestamp. How is that "backwards"?
Remember, by default, Splunk displays the newest events first.

---
If this reply helps you, Karma would be appreciated.
0 Karma

johnny21
Path Finder

Hey Rich - that is correct, I didn't want to conflate the issue as "technically" the props.conf as answered in that question addresses that one issue...it just also introduces a new one. I will try to clarify:

I am attaching a screenshot of what I mean by going backwards in the answers section below (as I can't attach screenshots in replies).

The blue line represents 11:59:59 then the red line takes us back to a previous date and inserts events from 12:00:00 and on to a previous day. The newest events are not being shown first; the newest event is at the end of the second red arrow.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...