Getting Data In

Why are multiple events being merged together as a single event?

Path Finder

Hi,

I have following logs which are being merged as a single event in Splunk. There are similar events which are correctly parsed by Splunk, but some of them are not.

2016-01-27 15:55:49.009064+00:00 name="http" event_id=http:notifier src=xx.xx.xx.xx dest="xx" dest_port=tcp/80 url="/xx" http_method="GET" http_user_agent="xx" http_refer="xx"
2016-01-27 15:55:49.016709+00:00 name="http" event_id=http:notifier src=xx dest="xx" dest_port=tcp/80 url="xx" http_method="GET" http_user_agent="xx" http_refer="xx"

and following is the props.conf for this:

[source::/var/log/abc/...]
TRANSFORMS-SK=a,b,c
TIME_PREFIX=^
TIME_FORMAT=%y-%m-%d %H:%M:%S.%6N%z
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = false

Any ideas on this?

0 Karma
Highlighted

Re: Why are multiple events being merged together as a single event?

Motivator

Try adding this to your props.conf

BREAK_ONLY_BEFORE = \d{4}-\d{2}-\d{2}\s
SHOULD_LINEMERGE = true
0 Karma
Highlighted

Re: Why are multiple events being merged together as a single event?

Path Finder

Should this work even on my indexed fields ? because currently it is not

0 Karma
Highlighted

Re: Why are multiple events being merged together as a single event?

Path Finder

** indexed events I meant above.

0 Karma
Highlighted

Re: Why are multiple events being merged together as a single event?

Motivator

What was already bad indexed wont get re-parsed. You should probably delete those and re-index.

0 Karma
Highlighted

Re: Why are multiple events being merged together as a single event?

Path Finder

As the issue is not replication too much, so cannot test it. But i didnot get the issue itself as these events are well formed with breaklines \r\n. Splunk can learn these events breaks automatically. Also why is it occurrence in some events and not others.

0 Karma
Highlighted

Re: Why are multiple events being merged together as a single event?

Motivator

Can you paste another data sample of those events that are not getting parsed correctly ?

Run this at CLI and search for the affected sourcetype stanza to debug.

 %SPLUNK_HOME$/bin>./splunk cmd btool props list
0 Karma
Highlighted

Re: Why are multiple events being merged together as a single event?

Path Finder

well nothing wrong i can found in props for the given sourcetype.

0 Karma
Highlighted

Re: Why are multiple events being merged together as a single event?

Motivator

Just use this on your props.conf & btw I had a typo on my first answer and already fixed it. This should work as expected.

[source::/var/log/abc/...]
 BREAK_ONLY_BEFORE = \d{4}-\d{2}-\d{2}\s
 SHOULD_LINEMERGE = true
 TIME_PREFIX=^
 TIME_FORMAT=%y-%m-%d %H:%M:%S.%6N%z
 MAX_TIMESTAMP_LOOKAHEAD = 32

Hope this is clear now.

0 Karma