All Apps and Add-ons

Splunk Add-on for JBoss: Why is Splunk combining log entries into one log entry?

cboillot
Contributor

We are importing logs from a JBoss server (with Splunk Add-on for JBoss installed), and we are noticing that there are several instances where there a few log entries combined into one log entry in Splunk. What would be causing this and how do I fix it? Is this a Splunk issue or an add-on issue?

0 Karma
1 Solution

rmills1
Explorer

I ran into the same problem, so I figured I'd post my fix here. Skoelpin was right about the line breaking problem. The line breaking and field extraction in the default props.conf doesn't work correctly (at least with my version of jboss). It's expecting a time field without a date, but my jboss logs had both date and time at the beginning of each line.

Here's the format in my logs: 2018-03-29 16:20:31,058

To fix it, put the following in $SPLUNK_HOME/etc/apps//local/props.conf:

[jboss:server:log]
EXTRACT-server = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\s+(?P<log_level>\w+)\s+\[(?P<event_category>[\-\.\w$]+)\]\s(?P<message>[\s\S]+)$
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}

View solution in original post

0 Karma

rmills1
Explorer

I ran into the same problem, so I figured I'd post my fix here. Skoelpin was right about the line breaking problem. The line breaking and field extraction in the default props.conf doesn't work correctly (at least with my version of jboss). It's expecting a time field without a date, but my jboss logs had both date and time at the beginning of each line.

Here's the format in my logs: 2018-03-29 16:20:31,058

To fix it, put the following in $SPLUNK_HOME/etc/apps//local/props.conf:

[jboss:server:log]
EXTRACT-server = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\s+(?P<log_level>\w+)\s+\[(?P<event_category>[\-\.\w$]+)\]\s(?P<message>[\s\S]+)$
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}
0 Karma

rmills1
Explorer

I noticed I accidentally hit the quote button instead of the code button, so the prop.conf lines were incorrect. Fixed to show correctly.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Your on the right track, but LINE_BREAKER is a better attribute than BREAK_ONLY_BEFORE and you should also set SHOULD_LINEMERGE=false

0 Karma

rmills1
Explorer

I based the attributes I used on the ones in the default props.conf. I wanted to modify it as little as possible. However, I might look into those attributes in addition.

Here's the default props.conf from the add-on:
[jboss:server:log]
MAX_TIMESTAMP_LOOKAHEAD = 32

01:59:41,057

EXTRACT-server = ^\d{2}:\d{2}:\d{2},\d{3}\s+(?P\w+)\s+[(?P[-.\w$]+)]\s((?P.+?))\s(?P[\s\S]+)$
BREAK_ONLY_BEFORE = ^\d{2}:\d{2}:\d{2},\d{3}
LOOKUP-severity_name = jboss_severity_lookup log_level OUTPUT severity
FIELDALIAS-body = message AS body

FIELDALIAS-subject = event_category AS subject

EVAL-app = "JBoss"

EDIT: The code block is apparently not working for me in comments section...ugg

0 Karma

skoelpin
SplunkTrust
SplunkTrust

It sounds like your events are not correctly line breaking. Can you go to $SPLUNK_HOME/etc/apps/<JBOSS_APP>/local and paste the contents of your props.conf? You should also provide a small set of sample data.

0 Karma

cboillot
Contributor

Um... I am not seeing a props.conf file.

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...