Getting Data In

Why is my props.conf configuration not parsing JBoss GC logs properly?

rzilist
Explorer

Hi guys,

I'm trying to parse GC logs from JBoss which look like this:

{Heap before GC invocations=1 (full 0):
 PSYoungGen      total 3670016K, used 692062K [0x0000000700000000, 0x0000000800000000, 0x0000000800000000)
  eden space 3145728K, 22% used [0x0000000700000000,0x000000072a3d7898,0x00000007c0000000)
  from space 524288K, 0% used [0x00000007e0000000,0x00000007e0000000,0x0000000800000000)
  to   space 524288K, 0% used [0x00000007c0000000,0x00000007c0000000,0x00000007e0000000)
 PSOldGen        total 8388608K, used 0K [0x0000000500000000, 0x0000000700000000, 0x0000000700000000)
  object space 8388608K, 0% used [0x0000000500000000,0x0000000500000000,0x0000000700000000)
 PSPermGen       total 786432K, used 37693K [0x00000004c0000000, 0x00000004f0000000, 0x0000000500000000)
  object space 786432K, 4% used [0x00000004c0000000,0x00000004c24cf628,0x00000004f0000000)
2015-01-11T01:12:55.373-0500: 8.667: [GC [PSYoungGen: 692062K->12100K(3670016K)] 692062K->12100K(12058624K), 0.0244480 secs] [Times: user=0.07 sys=0.00, real=0.02 secs] 
Heap after GC invocations=1 (full 0):
 PSYoungGen      total 3670016K, used 12100K [0x0000000700000000, 0x0000000800000000, 0x0000000800000000)
  eden space 3145728K, 0% used [0x0000000700000000,0x0000000700000000,0x00000007c0000000)
  from space 524288K, 2% used [0x00000007c0000000,0x00000007c0bd1290,0x00000007e0000000)
  to   space 524288K, 0% used [0x00000007e0000000,0x00000007e0000000,0x0000000800000000)
 PSOldGen        total 8388608K, used 0K [0x0000000500000000, 0x0000000700000000, 0x0000000700000000)
  object space 8388608K, 0% used [0x0000000500000000,0x0000000500000000,0x0000000700000000)
 PSPermGen       total 786432K, used 37693K [0x00000004c0000000, 0x00000004f0000000, 0x0000000500000000)
  object space 786432K, 4% used [0x00000004c0000000,0x00000004c24cf628,0x00000004f0000000)
}
{Heap before GC invocations=2 (full 1):
  ..................................................................
}

JBoss event starts with “{Heap before GC” and ends with “}” right before the next “{Heap before” stanza. Timestamp is buried in the middle. This is my props stanza (doesn't work):

TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 32
TZ = GMT
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
LINE_BREAKER=([\r\n]+){Heap  before GC
SHOULD_LINEMERGE = False
TRUNCATE = 10000
MAX_EVENTS = 256

What am I doing wrong here?

Thanks, Roman

Tags (3)
1 Solution

sk314
Builder

I suggest you try SHOULD_LINEMERGE = True, BREAK_ONLY_BEFORE = {Heap before GC

Also, remove TIME_PREFIX = ^ and MAX_TIMESTAMP_LOOKAHEAD = 32 and LINE_BREAKER = ([\r\n]+){Heap before GC (remove these three)

View solution in original post

sk314
Builder

I suggest you try SHOULD_LINEMERGE = True, BREAK_ONLY_BEFORE = {Heap before GC

Also, remove TIME_PREFIX = ^ and MAX_TIMESTAMP_LOOKAHEAD = 32 and LINE_BREAKER = ([\r\n]+){Heap before GC (remove these three)

rzilist
Explorer

Thank you sk314, works great! Splunk still complains about strptime for the first line ({Heap before GC invocations), but I guess that's because timestamp is buried in the middle of the event instead of being the first line.

How do I flag it as useful / correct answered?

Thank you
Roman

0 Karma

sk314
Builder

I think you can convert comment to answer, or upvote comment. (If it helped that is...)

0 Karma

rzilist
Explorer

Hi sk314, it was your answer that helped me, thank you!

0 Karma

rzilist
Explorer

ok, on the second thought it doesn't work in dev env when reading the same log file: looks fine in Data Preview locally, but in dev, Splunk splits event on the timestamp. I use 6.2 locally, and 6.1.4 in dev.

Thanks, Roman

Can anyone help with this?

Got it to work using same props file: restarted agent and re-read new logs. This is the final props:

TIME_PREFIX = ^

MAX_TIMESTAMP_LOOKAHEAD = 32

TZ = GMT
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z

LINE_BREAKER=([\r\n]+){Heap before GC

BREAK_ONLY_BEFORE = {Heap before GC
SHOULD_LINEMERGE = True
TRUNCATE = 10000
MAX_EVENTS = 256

Thank you to all who helped!

0 Karma

sk314
Builder

Try SHOULD_LINEMERGE = True.

0 Karma

rzilist
Explorer

hi sk314,
Thanks for the suggestion, same results. With LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}, Splunk breaks on timestamp splittign event in half on the date. I just want to break before "{Heap before GC" string

Thanks, Roman

0 Karma

rzilist
Explorer

Hi ppablo,

Sorry for the confusion: for this effort I'm not using Add-on, writing my own props.conf

0 Karma

ppablo
Retired

Hi @rzilist

No problem, thanks for making the edit. Also, when responding to people's answers or comments, be sure to type in the "Add your comment" box right below their answer/comment. You accidentally typed in the "Enter your answer here..." box at the very bottom of the page. I just converted it to a comment for you, just something to keep in mind from here on out. Thanks!

Patrick

0 Karma

rzilist
Explorer

Hi Patrick, sorry about that yet again: there's always a first time
Roman

0 Karma

ppablo
Retired

Hi @rzilist

I just wanted to clarify, but are you using the Add-on for JBoss: https://apps.splunk.com/app/1804/

You tagged it in your post (bluish green tag at the bottom of your post), but didn't mention it anywhere in your content. Just wanted to make sure it's tagged appropriately.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...