Getting Data In

Why are separate events combined as a single event from the 1st to the 9th of every month, but are parsed correctly as individual events starting on the 10th?

New Member

From the 1st until the 9th 23:59:59 of every month, individual events are being combined into one event. As soon as time shifts to the 10th 00:00:00, every events starts getting parsed properly with proper breaks before every date and time...Why is this happening, and how should I troubleshoot a fix for this?

Event 1)
Sep 10 00:00:00 10.39.246.155 date=2017-09-09 time=17:03:41 devname......and so on (single line)

Event 2)
Sep 10 00:00:00 10.39.246.155 date=2017-09-09 time=17:03:41 devname.......and so on(single line)

Event 3)
Sep 9 00:00:01 10.39.246.155 date=2017-09-08 time=17:03:26 devname........and so on (10 lines approx)
Sep 9 00:00:01 10.39.246.155 date=2017-09-08 time=17:03:26 devname.........and so on
Sep 9 00:00:01 10.39.246.155 date=2017-09-08 time=17:03:26 devname..........and so on

Event 4)
Sep 9 00:00:02 10.39.246.155 date=2017-09-08 time=17:03:26 devname.........and so on(10 lines approx)
Sep 9 00:00:01 10.39.246.155 date=2017-09-08 time=17:03:26 devname..........and so on

0 Karma

New Member

Hey,
We add below attribute for single and double digit date parsing and it works fine.
LINE_BREAKER = ([\n\r]+)\w+\s*(\d{1}|\d{2})\s\d{2}:\d{2}:\d{2}\s

Thanks for help..

0 Karma

New Member

Hey, issue resolved.
LINE_BREAKER = ([\n\r]+)\w+\s*(\d{1}|\d{2})\s\d{2}:\d{2}:\d{2}\s
finally above breaker helped us.

0 Karma

Splunk Employee
Splunk Employee

Sounds like your TIME_FORMAT specification does not cover single-digit day numbers (%e vs. %d).
If you didn't explicitly configure TIME_FORMAT for this sourcetype, I would recommend you do so.
It may also be your BREAK_ONLY_BEFORE pattern, hard to say without seeing your props.conf for this sourcetype.

But do share your props.conf if it still doesn't make sense, please.

0 Karma

New Member

Hi Ssievert,

We have tried below TIME_FORMAT.
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ([\n\r]*)\w{3}\s+\d+\s+\d{2}:\d{2}:\d{2}
NO_BINARY_CHECK = true
MAX_TIMESTAMP_LOOKAHEAD = 20

0 Karma

Splunk Employee
Splunk Employee

Yes, so you are using %d for the day, which expects two digits, i.e. 01...09 for day of month <10.
If you change it to %e things should look better (for newly indexed data only).
See here for a reference.

0 Karma

New Member

We have changed it to = TIME_FORMAT = %b %e %H:%M:%S.
But still same error. Do we need to change anything except this in above stanza?

0 Karma

Splunk Employee
Splunk Employee

OK, assuming you have these settings configured in the right place (indexer or heavy forwarder), this is what you should need to make it right:

TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S.%3N
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
MAX_TIMESTAMP_LOOKAHEAD = 30

If you don't have multi-line events, always set SHOULD_LINEMERGE to false. For single line events, the default event breaking should work just fine.

0 Karma

New Member

We have this stanza set on our indexer.
All attributes are placed as above.
Still getting merged events.
Anyway will check everything again tomorrow and post it here if issue gets fix.
Thanks for your help 🙂

0 Karma

New Member

This works fine for date parsing.
LINE_BREAKER = ([\n\r]+)\w+\s*(\d{1}|\d{2})\s\d{2}:\d{2}:\d{2}\s

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!