Why is my props.conf configuration not parsing JBoss GC logs properly?

[rzilist]

Hi guys,

I'm trying to parse GC logs from JBoss which look like this:

{Heap before GC invocations=1 (full 0):
 PSYoungGen      total 3670016K, used 692062K [0x0000000700000000, 0x0000000800000000, 0x0000000800000000)
  eden space 3145728K, 22% used [0x0000000700000000,0x000000072a3d7898,0x00000007c0000000)
  from space 524288K, 0% used [0x00000007e0000000,0x00000007e0000000,0x0000000800000000)
  to   space 524288K, 0% used [0x00000007c0000000,0x00000007c0000000,0x00000007e0000000)
 PSOldGen        total 8388608K, used 0K [0x0000000500000000, 0x0000000700000000, 0x0000000700000000)
  object space 8388608K, 0% used [0x0000000500000000,0x0000000500000000,0x0000000700000000)
 PSPermGen       total 786432K, used 37693K [0x00000004c0000000, 0x00000004f0000000, 0x0000000500000000)
  object space 786432K, 4% used [0x00000004c0000000,0x00000004c24cf628,0x00000004f0000000)
2015-01-11T01:12:55.373-0500: 8.667: [GC [PSYoungGen: 692062K->12100K(3670016K)] 692062K->12100K(12058624K), 0.0244480 secs] [Times: user=0.07 sys=0.00, real=0.02 secs] 
Heap after GC invocations=1 (full 0):
 PSYoungGen      total 3670016K, used 12100K [0x0000000700000000, 0x0000000800000000, 0x0000000800000000)
  eden space 3145728K, 0% used [0x0000000700000000,0x0000000700000000,0x00000007c0000000)
  from space 524288K, 2% used [0x00000007c0000000,0x00000007c0bd1290,0x00000007e0000000)
  to   space 524288K, 0% used [0x00000007e0000000,0x00000007e0000000,0x0000000800000000)
 PSOldGen        total 8388608K, used 0K [0x0000000500000000, 0x0000000700000000, 0x0000000700000000)
  object space 8388608K, 0% used [0x0000000500000000,0x0000000500000000,0x0000000700000000)
 PSPermGen       total 786432K, used 37693K [0x00000004c0000000, 0x00000004f0000000, 0x0000000500000000)
  object space 786432K, 4% used [0x00000004c0000000,0x00000004c24cf628,0x00000004f0000000)
}
{Heap before GC invocations=2 (full 1):
  ..................................................................
}

JBoss event starts with “{Heap before GC” and ends with “}” right before the next “{Heap before” stanza. Timestamp is buried in the middle. This is my props stanza (doesn't work):

TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 32
TZ = GMT
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
LINE_BREAKER=([\r\n]+){Heap  before GC
SHOULD_LINEMERGE = False
TRUNCATE = 10000
MAX_EVENTS = 256

What am I doing wrong here?

Thanks, Roman

[sk314 · Accepted solution]

I suggest you try SHOULD_LINEMERGE = True, BREAK_ONLY_BEFORE = {Heap before GC

Also, remove TIME_PREFIX = ^ and MAX_TIMESTAMP_LOOKAHEAD = 32 and LINE_BREAKER = ([\r\n]+){Heap before GC (remove these three)

[rzilist]

Thank you sk314, works great! Splunk still complains about strptime for the first line ({Heap before GC invocations), but I guess that's because timestamp is buried in the middle of the event instead of being the first line.

How do I flag it as useful / correct answered?

Thank you
Roman

[sk314]

I think you can convert comment to answer, or upvote comment. (If it helped that is...)

[rzilist]

Hi sk314, it was your answer that helped me, thank you!

[rzilist]

ok, on the second thought it doesn't work in dev env when reading the same log file: looks fine in Data Preview locally, but in dev, Splunk splits event on the timestamp. I use 6.2 locally, and 6.1.4 in dev.

Thanks, Roman

Can anyone help with this?

Got it to work using same props file: restarted agent and re-read new logs. This is the final props:

TIME_PREFIX = ^

MAX_TIMESTAMP_LOOKAHEAD = 32

TZ = GMT
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z

LINE_BREAKER=([\r\n]+){Heap before GC

BREAK_ONLY_BEFORE = {Heap before GC
SHOULD_LINEMERGE = True
TRUNCATE = 10000
MAX_EVENTS = 256

Thank you to all who helped!

[sk314]

Try SHOULD_LINEMERGE = True.

[rzilist]

hi sk314,
Thanks for the suggestion, same results. With LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}, Splunk breaks on timestamp splittign event in half on the date. I just want to break before "{Heap before GC" string

Thanks, Roman

[rzilist]

Hi ppablo,

Sorry for the confusion: for this effort I'm not using Add-on, writing my own props.conf

[ppablo]

Hi @rzilist

No problem, thanks for making the edit. Also, when responding to people's answers or comments, be sure to type in the "Add your comment" box right below their answer/comment. You accidentally typed in the "Enter your answer here..." box at the very bottom of the page. I just converted it to a comment for you, just something to keep in mind from here on out. Thanks!

Patrick

[rzilist]

Hi Patrick, sorry about that yet again: there's always a first time
Roman

[ppablo]

Hi @rzilist

I just wanted to clarify, but are you using the Add-on for JBoss: https://apps.splunk.com/app/1804/

You tagged it in your post (bluish green tag at the bottom of your post), but didn't mention it anywhere in your content. Just wanted to make sure it's tagged appropriately.