Log directly to Splunk with log4j

[tbo]

Hi,
I'm doing some testing on how to use Splunk the best possible way. I have fallen in love with the method of using the Rest API and log4j, since it doesn't require any forwarder or anything to be installed. Anyway, I have written a test application. It consists of a class and a log4j-config file:

AutoLogger:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import org.slf4j.*;
import com.splunk.logging.SplunkLogEvent;

public class AutoLogger {

    public static void main(String[] args) throws IOException {
        Logger logger = LoggerFactory.getLogger("splunk.logger");

        InputStreamReader converter = new InputStreamReader(System.in);
        BufferedReader reader = new BufferedReader(converter);

        String in = "";
        while(!in.equals("exit")) {
            System.out.println("Log something:");
            in = reader.readLine();

            SplunkLogEvent event = new SplunkLogEvent("User logged something", "user_log");
            event.addPair("input", in);

            logger.error(event.toString());
        }
    }
}

log4j.properties: (stolen from https://github.com/splunk/splunk-library-javalogging)

#Example log4j configuration using a Splunk REST Appender or Splunk Raw TCP Appender

# Root logger option
log4j.rootLogger=DEBUG,stdout

#Splunk logger
log4j.logger.splunk.logger=INFO, splunkrest
#log4j.logger.splunk.logger=INFO, splunkrawtcp
log4j.additivity.splunk.logger=false

# Send log events to a Splunk REST endpoint
log4j.appender.splunkrest=com.splunk.logging.log4j.appender.SplunkRestAppender
log4j.appender.splunkrest.user=admin
log4j.appender.splunkrest.pass=splunk
log4j.appender.splunkrest.host=splunkserver
log4j.appender.splunkrest.port=8089
log4j.appender.splunkrest.delivery=stream
log4j.appender.splunkrest.metaSource=rest
log4j.appender.splunkrest.metaSourcetype=testing
log4j.appender.splunkrest.metaIndex=main
log4j.appender.splunkrest.maxQueueSize=5MB
log4j.appender.splunkrest.dropEventsOnQueueFull=false
log4j.appender.splunkrest.layout=org.apache.log4j.PatternLayout
log4j.appender.splunkrest.layout.ConversionPattern=%m%n
# optionally you can enrich the messages with formatting tokens from the logging framework
#log4j.appender.splunkrest.layout.ConversionPattern=%d{ABSOLUTE} %m%n
#log4j.appender.splunkrest.layout.ConversionPattern=%m loglevel="%p"%n

# Send log events to a Splunk Raw TCP server socket
log4j.appender.splunkrawtcp=com.splunk.logging.log4j.appender.SplunkRawTCPAppender
log4j.appender.splunkrawtcp.host=splunkserver
log4j.appender.splunkrawtcp.port=5150
log4j.appender.splunkrawtcp.maxQueueSize=5MB
log4j.appender.splunkrawtcp.dropEventsOnQueueFull=false
log4j.appender.splunkrawtcp.layout=org.apache.log4j.PatternLayout
log4j.appender.splunkrawtcp.layout.ConversionPattern=%m%n
# optionally you can enrich the messages with formatting tokens from the logging framework
#log4j.appender.splunkrawtcp.layout.ConversionPattern=%d{ABSOLUTE} %m%n
#log4j.appender.splunkrawtcp.layout.ConversionPattern=%m loglevel="%p"%n

#Console appender
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L - %m%n

Running this application should allow the user to type in a line in the console, which should then be logged via log4j, and show up in Splunk. And this works. Sometimes! But only sometimes. Whether the logs arrive at Splunk or not seems completely random!

Does anyone have even the slightest idea about what can possibly be going wrong?

[Damien_Dallimor · Accepted solution]

With the REST appender , there are 2 modes , simple and stream.

log4j.appender.splunkrest.delivery=stream

log4j.appender.splunkrest.delivery=simple

The simple appender is ok for small number of messages , it will open a connection send the log event and then close the connection. So doesn't really scale so well.But messages will show up immediately in Splunk.

The stream appender scales better , but due to a quirk in the REST endpoint logic for the receivers/stream endpoint, events only show up in splunk when either a) you close the connection ie: terminate your test program or b) you send enough data through to cause the data receiving buffer in Splunk for the REST endpoint to flush.This needs to be remedied in Splunk.

My preference is however is to use the TCP appender.

[Damien_Dallimor]

See above regarding what I already wrote about "stream".

[pchethan]

@Damien, for me the logs via stream, do not show up at all. Is it usable at all? Are there any workarounds?

[Damien_Dallimor]

With TCP you need to setup the TCP input and metadata (sourcetype, index etc..) on the Splunk side.

With REST , you don't need to setup anything on the Splunk side as you can specify the source, sourcetype, index, host as part of the REST request.

Performance wise the REST stream and Raw TCP will be similar.

REST simple does not perform well under high load.

So, based on this info , choose the most appropriate use case for your environment.

[tbo]

Perfect! Exactly the answer I was looking for! Is there any situation wherein it would be preferable to use the REST appender?

[psanford_splunk]

We have taken logging framework repo from GitHub and have now also included it in our Plug-In for Eclipse. Details are here: http://dev.splunk.com/view/splunk-plugin-eclipse/SP-CAAAEQP

We also have a number of code templates to help get you started. Assuming you are okay with using eclipse, I would start there.

[tbo]

Thanks. That's the one I've been using. I have sort of-followed this guide (http://dev.splunk.com/view/splunk-plugin-eclipse/SP-CAAAEQY) (sort of, since as you can see my class are different and I'm using log4j). It still didn't work. As it seemed to be an issue with REST, I have now switched to using raw tcp, which seems to work very well. But it's hard to tell, since it was all very random before.

[koshyk]

You could log into a centralised log server (location) where splunk has access of. You need to take care of log rotation/archiving etc..

[tbo]

Update:

This might be of some help:
I looked at the "Splunk errors last 24 hours"-page, and a lot of these guys shows up:

ERROR HttpListener - Exception while processing request from MYIP for /services/receivers/stream?index=main&source=rest&sourcetype=testing: Timeout