Splunk Search

How to break Weblogic JVM events using regex with multiple timestamps & exceptions in single event

p_gurav
Champion

Hi All,

I have the following JVM logs:

May 8, 2016 1:26:26 AM IST Warning Socket BEA-000449 Closing socket as no data read from it on x.x.x.x:x,x during the configured idle timeout of 5 secs
01:45:05.078 [Listener-null] ERROR c.c.s.ListenerHandler - Error occurred while delivering response message for listener rilBPMService.createServiceOrderInterface.createOrder.
java.lang.NullPointerException
        at com.conceptwave.serviceprovider.MsgQueueListener$MsgQueueHandler.doCreateOutConnection(MsgQueueListener.java:250)
        at com.conceptwave.serviceprovider.ListenerHandler.createOutConnection(ListenerHandler.java:517)
        at com.conceptwave.serviceprovider.ListenerHandler.run(ListenerHandler.java:281)
        at com.conceptwave.serviceprovider.CwfListener.run(CwfListener.java:320)
02:00:12.712 [Listener-null] ERROR c.c.s.ListenerHandler - Error occurred while delivering response message for listener rilBPMService.createServiceOrderInterface.createOrder.
java.lang.NullPointerException
        at com.conceptwave.serviceprovider.MsgQueueListener$MsgQueueHandler.doCreateOutConnection(MsgQueueListener.java:250)
        at com.conceptwave.serviceprovider.ListenerHandler.createOutConnection(ListenerHandler.java:517)
        at com.conceptwave.serviceprovider.ListenerHandler.run(ListenerHandler.java:281)
        at com.conceptwave.serviceprovider.CwfListener.run(CwfListener.java:320)
02:00:14.008 [Listener-null] ERROR c.c.s.ListenerHandler - Error occurred while delivering response message for listener rilBPMService.serviceOrderNotificationInterface.serviceOrderNotification.
java.lang.NullPointerException
        at com.conceptwave.serviceprovider.MsgQueueListener$MsgQueueHandler.doCreateOutConnection(MsgQueueListener.java:250)
        at com.conceptwave.serviceprovider.ListenerHandler.createOutConnection(ListenerHandler.java:517)
        at com.conceptwave.serviceprovider.ListenerHandler.run(ListenerHandler.java:281)
        at com.conceptwave.serviceprovider.CwfListener.run(CwfListener.java:320)
02:00:46.377 [Listener-null] ERROR c.c.s.ListenerHandler - Error occurred while delivering response message for listener rilBPMService.createServiceOrderInterface.createOrder.
java.lang.NullPointerException
        at com.conceptwave.serviceprovider.MsgQueueListener$MsgQueueHandler.doCreateOutConnection(MsgQueueListener.java:250)
        at com.conceptwave.serviceprovider.ListenerHandler.createOutConnection(ListenerHandler.java:517)
        at com.conceptwave.serviceprovider.ListenerHandler.run(ListenerHandler.java:281)
        at com.conceptwave.serviceprovider.CwfListener.run(CwfListener.java:320)

I want Splunk to consider above event as one event. But as it has multiple timestamps without date.

I tried following properties in props.conf:

BREAK_ONLY_BEFORE = \<\w\w\w\s\d{2},\s\d{4}\s\d+:\d{2}:\d{2}\>
SHOULD_LINEMERGE = true
MAX_EVENTS= 10000
TIME_FORMAT=%b %d, %Y %H:%M:%S %p
TIME_PREFIX=^

In this case Splunk is breaking event whenever it encounters time (i.e. at each exception), creating multiple events for each exception instead of single event.

If I don't use the BREAK_ONLY_BEFORE in props, then Splunk is still breaking the event whenever it encounters time, but not indexing the exception part. I end up with missing data.

Could anybody will help me with this?
Thank You in advance

michael_sleep
Communicator

This would work as a props configuration. Just take whatever you need. The regex may be different if only because I can't tell if your timestamp line starts with a space or not, so I've included both. This one will assume there is no space:

SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^(\w{3}\s\d{1,2}\,\s\d{4}\s\d{1,2}:\d{2}:\d{2}\s\w{2}\s\w{3})
MAX_TIMESTAMP_LOOKAHEAD=27

If there is a space:

^(\s\w{3}\s\d{1,2}\,\s\d{4}\s\d{1,2}:\d{2}:\d{2}\s\w{2}\s\w{3})
0 Karma

campbellj1977
Explorer

It looks like you are defining your time prefix to start of new line, but no line breaks mentioned. Try this;

SHOULD_LINEMERGE = false
LINE_BREAKER =  \w{3}\s\w{1,2}\,\s\d{4}\s\d{1,2}\:\d{1,2}\:\d{1,2}\s\w{2}
TIME_FORMAT=%b %d, %Y %H:%M:%S %p
TIME_PREFIX=^
LOOK_AHEAD = 45

I believe this is best practice all props.

Thanks,
Joshua

0 Karma

woodcock
Esteemed Legend

Try this:

TIME_PREFIX = ^
TIME_FORMAT = %b %d, %Y %H:%M:%S %p
BREAK_ONLY_BEFORE_DATE = true
SHOULD_LINEMERGE = true
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...