Splunk Add-on for JBoss: Why is Splunk combining log entries into one log entry?

[cboillot]

We are importing logs from a JBoss server (with Splunk Add-on for JBoss installed), and we are noticing that there are several instances where there a few log entries combined into one log entry in Splunk. What would be causing this and how do I fix it? Is this a Splunk issue or an add-on issue?

[rmills1 · Accepted solution]

I ran into the same problem, so I figured I'd post my fix here. Skoelpin was right about the line breaking problem. The line breaking and field extraction in the default props.conf doesn't work correctly (at least with my version of jboss). It's expecting a time field without a date, but my jboss logs had both date and time at the beginning of each line.

Here's the format in my logs: 2018-03-29 16:20:31,058

To fix it, put the following in $SPLUNK_HOME/etc/apps//local/props.conf:

[jboss:server:log]
EXTRACT-server = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\s+(?P<log_level>\w+)\s+\[(?P<event_category>[\-\.\w$]+)\]\s(?P<message>[\s\S]+)$
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}

[rmills1]

I noticed I accidentally hit the quote button instead of the code button, so the prop.conf lines were incorrect. Fixed to show correctly.

[skoelpin]

Your on the right track, but LINE_BREAKER is a better attribute than BREAK_ONLY_BEFORE and you should also set SHOULD_LINEMERGE=false

[rmills1]

I based the attributes I used on the ones in the default props.conf. I wanted to modify it as little as possible. However, I might look into those attributes in addition.

Here's the default props.conf from the add-on:
[jboss:server:log]
MAX_TIMESTAMP_LOOKAHEAD = 32

01:59:41,057

EXTRACT-server = ^\d{2}:\d{2}:\d{2},\d{3}\s+(?P\w+)\s+[(?P[-.\w$]+)]\s((?P.+?))\s(?P[\s\S]+)$
BREAK_ONLY_BEFORE = ^\d{2}:\d{2}:\d{2},\d{3}
LOOKUP-severity_name = jboss_severity_lookup log_level OUTPUT severity
FIELDALIAS-body = message AS body

FIELDALIAS-subject = event_category AS subject

EVAL-app = "JBoss"

EDIT: The code block is apparently not working for me in comments section...ugg

[skoelpin]

It sounds like your events are not correctly line breaking. Can you go to $SPLUNK_HOME/etc/apps/<JBOSS_APP>/local and paste the contents of your props.conf? You should also provide a small set of sample data.

[cboillot]

Um... I am not seeing a props.conf file.