Log4j Splunk

[LauraBre]

Hello,

I work with Eclipse. I want to send my log on the port listened by Splunk. So I add this code in log4j.properties :

log4j.logger.net.awl.bfi.TokenizerWatchdogSplunk=info,watchdogSplunkSocket
log4j.appender.watchdogSplunkSocket=org.apache.log4j.net.SyslogAppender
log4j.appender.watchdogSplunkSocket.SyslogHost=    [:CustomPort]log4j.appender.watchdogSplunkSocket.remoteHost=localhost
log4j.appender.watchdogSplunkSocket.port=5540
log4j.appender.watchdogSplunkSocket.locationInfo=true
log4j.appender.watchdogSplunkSocket.layout=org.apache.log4j.PatternLayout 
log4j.appender.watchdogSplunkSocket.layout.ConversionPattern = [%-5p][%d{dd/MM/yyyy HH:mm:ss}][%c][%F]%m%n
log4j.appender.watchdogSplunkSocket.Facility=USER

I managed to recover the file in Splunk but there are a lot of caracters that to add in my log file. For example I had the following event in Splunk :

sr\x00!org.apache.log4j.spi.LoggingEvent\xF3\xF2\xB9#t\xB\xB5?\x3\x00Z\x00\x15mdcCopyLookupRequiredZ\x00\x11ndcLookupRequiredJ\x00  timeStampL\x00\xCcategoryNamet\x00\x12Ljava/lang/String;L\x00\xClocationInfot\x00#Lorg/apache/log4j/spi/LocationInfo;L\x00\x7mdcCopyt\x00\x15Ljava/util/Hashtable;L\x00\x3ndcq\x00~\x00\x1L\x00\xFrenderedMessageq\x00~\x00\x1L\x00threadNameq\x00~\x00\x1L\x00throwableInfot\x00+Lorg/apache/log4j/spi/ThrowableInformation;xp\x00\x00\x00\x00\x17Q7\xA6\xE0t\x00#net.awl.bfi.TokenizerWatchdogSplunksr\x00!org.apache.log4j.spi.LocationInfo홻\xE1J\x91\xA5|\x2\x00\x1L\x00\x8fullInfoq\x00~\x00\x1xpt\x00Dnet.awl.bfi.eipengine.beans.MapInfoBean.process(MapInfoBean.java:86)ppt\x00URequested serv. : D2T, Nb PAN : 1, Requester : a187571, User : a508404, Host : W60080t\x00\x15ActiveMQ Session Taskpw\x4\x00\x00N pxysr\x00!org.apache.log4j.spi.LoggingEvent\xF3\xF2\xB9#t\xB\xB5?\x3\x00Z\x00

But I just add Requested serv. : D2T, Nb PAN : 1, Requester : a187571, User : a508404, Host : W60080t in my log.

Can I resolve this problem???

[LauraBre]

I tried to use SplunkLogEvent but I don't able to listen the events in splunk and in local. I think there is a problem during the sending.

This is my log4j file :

#log4j.logger.splunk.logger=INFO, splunkrest
log4j.logger.splunk.logger=INFO, splunkrawtcp
log4j.additivity.splunk.logger=false
.
.
.   
# Send log events to a Splunk Raw TCP server socket
log4j.appender.splunkrawtcp=com.dtdsoftware.splunk.logging.log4j.appender.SplunkRawTCPAppender
log4j.appender.splunkrawtcp.host=odpcil01b
log4j.appender.splunkrawtcp.port=5543
log4j.appender.splunkrawtcp.maxQueueSize=5MB
log4j.appender.splunkrawtcp.dropEventsOnQueueFull=false
log4j.appender.splunkrawtcp.layout=org.apache.log4j.PatternLayout
log4j.appender.splunkrawtcp.layout.ConversionPattern=%m%n
# optionally you can enrich the messages with formatting tokens from the logging framework
#log4j.appender.splunkrawtcp.layout.ConversionPattern=%d{ABSOLUTE} %m%n
#log4j.appender.splunkrawtcp.layout.ConversionPattern=%m loglevel="%p"%n

For the example, I used the example of the link that you
gave me.

Thanks by advance.

[Damien_Dallimor]

If you are using log4j you can't even log locally(file or console), then perhaps you have either :
a) not setup your classpath for slf4j to use log4j correctly.You should only have the log4j bindings on your classpath, you don't need the logback(logback-classic.jar , logback-core.jar) and java.util(slf4j-jdk14-1.6.4.jar) bindings
b) log4j.properties is not on your classpath.

[Damien_Dallimor]

You have a TCP port open in Splunk and the log4j appender "org.apache.log4j.net.SyslogAppender" is UDP.

Also , your log4j.properties looks wrong.
SyslogAppender doesn't use the remoteHost and port properties.
It use the syslogHost property

Suggested Corrections :

log4j.properties

log4j.logger.net.awl.bfi.TokenizerWatchdogSplunk=info,watchdogSplunkSocket log4j.appender.watchdogSplunkSocket=org.apache.log4j.net.SyslogAppender log4j.appender.watchdogSplunkSocket.SyslogHost=odpcil01b:5541
log4j.appender.watchdogSplunkSocket.locationInfo=false log4j.appender.watchdogSplunkSocket.layout=org.apache.log4j.PatternLayout log4j.appender.watchdogSplunkSocket.layout.ConversionPattern = [%-5p][%d{dd/MM/yyyy HH:mm:ss}][%c][%F]%m%n log4j.appender.watchdogSplunkSocket.Facility=USER

inputs.conf

[udp://5541] 
connection_host = none 
sourcetype = log4j

The Log4j SocketAppender won't work because that appender sends a serialized object(binary)

If you do want to use TCP , then check out SplunkJavaLogging

• Log4j and Logback appenders to send events to Splunk via HTTP REST or Raw TCP
• Helper classes for formatting log events in a best practice semantic format for Splunk

[Damien_Dallimor]

If using the Raw TCP Appender from SplunkJavaLogging , it's up to you how you implement your logging code. My preference is to use the slf4j logging facade to abstract the underlying logging implementation (log4j, logback, java.util.logging).And the SplunkLogEvent class is useful for creating your log events in a best practice semantic format.
If using slf4j ontop of log4j, you then place the necesssary delegate jars on the classpath(ie:slf4j-log4j12-1.6.4.jar and log4j-1.2.16.jar ) and configure the properties file(ie: log4j.properties).

[LauraBre]

Ok thanks. If I wang to use TCP, I just to import org.slf4j.LoggerFactory to format my reglar string log to index correctly in splunk.

[Damien_Dallimor]

UDP does not guarantee delivery.

[LauraBre]

Thanks very much but I have a question: When we use UDP, we can lost data packets???

[LauraBre]

log4j.properties :

log4j.logger.net.awl.bfi.TokenizerWatchdogSplunk=info,watchdogSplunkSocket
log4j.appender.watchdogSplunkSocket=org.apache.log4j.net.SyslogAppender
log4j.appender.watchdogSplunkSocket.remoteHost=odpcil01b
log4j.appender.watchdogSplunkSocket.port=5541
log4j.appender.watchdogSplunkSocket.locationInfo=false
log4j.appender.watchdogSplunkSocket.layout=org.apache.log4j.PatternLayout
log4j.appender.watchdogSplunkSocket.layout.ConversionPattern = [%-5p][%d{dd/MM/yyyy HH:mm:ss}][%c][%F]%m%n
log4j.appender.watchdogSplunkSocket.Facility=USER

I don't know if I have to use a SylogAppender or SocketAppender.

I am logging in my Java code as this:
LOGGER_SPLUNK.info("Requested serv. : D2T, Nb PAN : " + nbPan +", Requester : " + body.get(NlvValue.REQUESTER_ID) +", User : " + body.get(NlvValue.USER_ID)+", Host : "+host+" ServiceName : ");

inputs.conf file :

[tcp://5541]
connection_host = none
sourcetype = log4j

Thanks by advance,

Laura

[Damien_Dallimor]

For some reason you have sent a serialized Java object (binary data) to Splunk rather than a textual log message.What does your logging statement in your code look like ?

You might want to check out SplunkJavaLogging , available on Github or via dev.splunk.com

Here you will find appenders for log4j and logback that use HTTP REST to send log events to Splunk.

There are also other utilities to help you format log messages into Splunk best practice logging format and use the Splunk Common Information Model (CIM)

[Damien_Dallimor]

I'll need to see :
1) your full log4j.properties file
2) an example of how you are logging in your Java code
3) the inputs.conf stanza for the UDP input in Splunk

Mask out any sensitive/confidential information.

[LauraBre]

In my log lig, I only have messages of this type :
Requested serv. : D2T, Nb PAN : 1, Requester : a187571, User : a508404, Host : W60080t
When I index directly this file from Splunk, it's perfect. By when I use a port, it's bad. But in eclipse, I use "ganymede Log4j View" to look that there are in the log on the port. When I look it, I see the message in correct format.

[Ayn]

Not very familiar with log4j, but the output that reaches Splunk seems to be in some kind of binary format rather than just plain ascii. You'll probably need to configure log4j to get a valid output format.