topic Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th? in Getting Data In

Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

rahulcrest — Thu, 02 Nov 2017 16:27:58 GMT

From the 1st until the 9th 23:59:59 of every month, individual events are being combined into one event. As soon as time shifts to the 10th 00:00:00, every events starts getting parsed properly with proper breaks before every date and time...Why is this happening, and how should I troubleshoot a fix for this?

Event 1)
Sep 10 00:00:00 10.39.246.155 date=2017-09-09 time=17:03:41 devname......and so on (single line)

Event 2)
Sep 10 00:00:00 10.39.246.155 date=2017-09-09 time=17:03:41 devname.......and so on(single line)

Event 3)
Sep 9 00:00:01 10.39.246.155 date=2017-09-08 time=17:03:26 devname........and so on (10 lines approx)
Sep 9 00:00:01 10.39.246.155 date=2017-09-08 time=17:03:26 devname.........and so on
Sep 9 00:00:01 10.39.246.155 date=2017-09-08 time=17:03:26 devname..........and so on

Event 4)
Sep 9 00:00:02 10.39.246.155 date=2017-09-08 time=17:03:26 devname.........and so on(10 lines approx)
Sep 9 00:00:01 10.39.246.155 date=2017-09-08 time=17:03:26 devname..........and so on

Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

s2_splunk — Tue, 29 Sep 2020 16:33:01 GMT

Sounds like your TIME_FORMAT specification does not cover single-digit day numbers (%e vs. %d).
If you didn't explicitly configure TIME_FORMAT for this sourcetype, I would recommend you do so.
It may also be your BREAK_ONLY_BEFORE pattern, hard to say without seeing your props.conf for this sourcetype.

But do share your props.conf if it still doesn't make sense, please.

Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

rahulcrest — Tue, 29 Sep 2020 16:36:27 GMT

Hi Ssievert,

We have tried below TIME_FORMAT.
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ([\n\r]*)\w{3}\s+\d+\s+\d{2}:\d{2}:\d{2}
NO_BINARY_CHECK = true
MAX_TIMESTAMP_LOOKAHEAD = 20

Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

s2_splunk — Thu, 02 Nov 2017 17:23:49 GMT

Yes, so you are using %d for the day, which expects two digits, i.e. 01...09 for day of month <10.
If you change it to %e things should look better (for newly indexed data only).
See here for a reference.

Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

rahulcrest — Thu, 02 Nov 2017 17:30:19 GMT

We have changed it to = TIME_FORMAT = %b %e %H:%M:%S.
But still same error. Do we need to change anything except this in above stanza?

Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

s2_splunk — Tue, 29 Sep 2020 16:33:07 GMT

OK, assuming you have these settings configured in the right place (indexer or heavy forwarder), this is what you should need to make it right:

TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S.%3N
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
MAX_TIMESTAMP_LOOKAHEAD = 30

If you don't have multi-line events, always set SHOULD_LINEMERGE to false. For single line events, the default event breaking should work just fine.

Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

rahulcrest — Thu, 02 Nov 2017 18:34:41 GMT

We have this stanza set on our indexer.
All attributes are placed as above.
Still getting merged events.
Anyway will check everything again tomorrow and post it here if issue gets fix.
Thanks for your help 🙂

Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

rahulcrest — Fri, 03 Nov 2017 10:07:47 GMT

Hey, issue resolved.
LINE_BREAKER = ([\n\r]+)\w+\s*(\d{1}|\d{2})\s\d{2}:\d{2}:\d{2}\s
finally above breaker helped us.

Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

rahulcrest — Fri, 03 Nov 2017 10:46:47 GMT

Hey,
We add below attribute for single and double digit date parsing and it works fine.
LINE_BREAKER = ([\n\r]+)\w+\s*(\d{1}|\d{2})\s\d{2}:\d{2}:\d{2}\s

Thanks for help..

Re: Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

rahulcrest — Fri, 03 Nov 2017 10:48:02 GMT

This works fine for date parsing.
LINE_BREAKER = ([\n\r]+)\w+\s*(\d{1}|\d{2})\s\d{2}:\d{2}:\d{2}\s