How to debug multi-line events not working

jeffwarn · ‎08-08-2013

I'm trying to determine why multiline events are not working when syslog sends the data over to my splunk indexer. The servers are setup to use log4j and send data to both the console (nfs) and syslog.

I took the event from the console/nfs log and put it into a text file on the same system and used netcat to send the event to the splunk indexer and it worked just fine.

Is there any way to determine what the problem is when the syslog server is sending the event? It's basically just creating single events for each line.

Here the code multiline event:

2013-08-08 16:20:28,845 [instance=testserver.i1] [tomcat-http--49] INFO org.apache.cxf.interceptor.LoggingOutInterceptor (AbstractLoggingInterceptor.java:149) - Outbound Message

ID: 190
Response-Code: 200
Content-Type: application/xml
Headers: {Date=[Thu, 08 Aug 2013 20:20:28 GMT]}
Payload: One line of data
Another line of data
data
data
data
data
data

davecroto · ‎08-08-2013

SHOULD_LINEMERGE = true

jeffwarn · ‎08-08-2013

I already have that configured. It works fine when read in via the file (or netcat). I'm not sure if it's something on the syslog of the app server or not. We have production systems that seem to be able to work fine with the settings I have. This is what I have in my config:

[source::udp:55514]
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = true
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 25