Getting Data In

Log directly to Splunk with log4j

tbo
Explorer

Hi,
I'm doing some testing on how to use Splunk the best possible way. I have fallen in love with the method of using the Rest API and log4j, since it doesn't require any forwarder or anything to be installed. Anyway, I have written a test application. It consists of a class and a log4j-config file:

AutoLogger:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import org.slf4j.*;
import com.splunk.logging.SplunkLogEvent;

public class AutoLogger {

    public static void main(String[] args) throws IOException {
        Logger logger = LoggerFactory.getLogger("splunk.logger");

        InputStreamReader converter = new InputStreamReader(System.in);
        BufferedReader reader = new BufferedReader(converter);

        String in = "";
        while(!in.equals("exit")) {
            System.out.println("Log something:");
            in = reader.readLine();

            SplunkLogEvent event = new SplunkLogEvent("User logged something", "user_log");
            event.addPair("input", in);

            logger.error(event.toString());
        }
    }
}

log4j.properties: (stolen from https://github.com/splunk/splunk-library-javalogging)

#Example log4j configuration using a Splunk REST Appender or Splunk Raw TCP Appender

# Root logger option
log4j.rootLogger=DEBUG,stdout

#Splunk logger
log4j.logger.splunk.logger=INFO, splunkrest
#log4j.logger.splunk.logger=INFO, splunkrawtcp
log4j.additivity.splunk.logger=false

# Send log events to a Splunk REST endpoint
log4j.appender.splunkrest=com.splunk.logging.log4j.appender.SplunkRestAppender
log4j.appender.splunkrest.user=admin
log4j.appender.splunkrest.pass=splunk
log4j.appender.splunkrest.host=splunkserver
log4j.appender.splunkrest.port=8089
log4j.appender.splunkrest.delivery=stream
log4j.appender.splunkrest.metaSource=rest
log4j.appender.splunkrest.metaSourcetype=testing
log4j.appender.splunkrest.metaIndex=main
log4j.appender.splunkrest.maxQueueSize=5MB
log4j.appender.splunkrest.dropEventsOnQueueFull=false
log4j.appender.splunkrest.layout=org.apache.log4j.PatternLayout
log4j.appender.splunkrest.layout.ConversionPattern=%m%n
# optionally you can enrich the messages with formatting tokens from the logging framework
#log4j.appender.splunkrest.layout.ConversionPattern=%d{ABSOLUTE} %m%n
#log4j.appender.splunkrest.layout.ConversionPattern=%m loglevel="%p"%n

# Send log events to a Splunk Raw TCP server socket
log4j.appender.splunkrawtcp=com.splunk.logging.log4j.appender.SplunkRawTCPAppender
log4j.appender.splunkrawtcp.host=splunkserver
log4j.appender.splunkrawtcp.port=5150
log4j.appender.splunkrawtcp.maxQueueSize=5MB
log4j.appender.splunkrawtcp.dropEventsOnQueueFull=false
log4j.appender.splunkrawtcp.layout=org.apache.log4j.PatternLayout
log4j.appender.splunkrawtcp.layout.ConversionPattern=%m%n
# optionally you can enrich the messages with formatting tokens from the logging framework
#log4j.appender.splunkrawtcp.layout.ConversionPattern=%d{ABSOLUTE} %m%n
#log4j.appender.splunkrawtcp.layout.ConversionPattern=%m loglevel="%p"%n

#Console appender
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L - %m%n

Running this application should allow the user to type in a line in the console, which should then be logged via log4j, and show up in Splunk. And this works. Sometimes! But only sometimes. Whether the logs arrive at Splunk or not seems completely random!

Does anyone have even the slightest idea about what can possibly be going wrong?

Tags (4)
0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

With the REST appender , there are 2 modes , simple and stream.

log4j.appender.splunkrest.delivery=stream

log4j.appender.splunkrest.delivery=simple

The simple appender is ok for small number of messages , it will open a connection send the log event and then close the connection. So doesn't really scale so well.But messages will show up immediately in Splunk.

The stream appender scales better , but due to a quirk in the REST endpoint logic for the receivers/stream endpoint, events only show up in splunk when either a) you close the connection ie: terminate your test program or b) you send enough data through to cause the data receiving buffer in Splunk for the REST endpoint to flush.This needs to be remedied in Splunk.

My preference is however is to use the TCP appender.

View solution in original post

Damien_Dallimor
Ultra Champion

With the REST appender , there are 2 modes , simple and stream.

log4j.appender.splunkrest.delivery=stream

log4j.appender.splunkrest.delivery=simple

The simple appender is ok for small number of messages , it will open a connection send the log event and then close the connection. So doesn't really scale so well.But messages will show up immediately in Splunk.

The stream appender scales better , but due to a quirk in the REST endpoint logic for the receivers/stream endpoint, events only show up in splunk when either a) you close the connection ie: terminate your test program or b) you send enough data through to cause the data receiving buffer in Splunk for the REST endpoint to flush.This needs to be remedied in Splunk.

My preference is however is to use the TCP appender.

Damien_Dallimor
Ultra Champion

See above regarding what I already wrote about "stream".

0 Karma

pchethan
New Member

@Damien, for me the logs via stream, do not show up at all. Is it usable at all? Are there any workarounds?

0 Karma

Damien_Dallimor
Ultra Champion

With TCP you need to setup the TCP input and metadata (sourcetype, index etc..) on the Splunk side.

With REST , you don't need to setup anything on the Splunk side as you can specify the source, sourcetype, index, host as part of the REST request.

Performance wise the REST stream and Raw TCP will be similar.

REST simple does not perform well under high load.

So, based on this info , choose the most appropriate use case for your environment.

tbo
Explorer

Perfect! Exactly the answer I was looking for! Is there any situation wherein it would be preferable to use the REST appender?

0 Karma

psanford_splunk
Splunk Employee
Splunk Employee

We have taken logging framework repo from GitHub and have now also included it in our Plug-In for Eclipse. Details are here: http://dev.splunk.com/view/splunk-plugin-eclipse/SP-CAAAEQP

We also have a number of code templates to help get you started. Assuming you are okay with using eclipse, I would start there.

0 Karma

tbo
Explorer

Thanks. That's the one I've been using. I have sort of-followed this guide (http://dev.splunk.com/view/splunk-plugin-eclipse/SP-CAAAEQY) (sort of, since as you can see my class are different and I'm using log4j). It still didn't work. As it seemed to be an issue with REST, I have now switched to using raw tcp, which seems to work very well. But it's hard to tell, since it was all very random before.

0 Karma

koshyk
Super Champion

You could log into a centralised log server (location) where splunk has access of. You need to take care of log rotation/archiving etc..

0 Karma

tbo
Explorer

Update:

This might be of some help:
I looked at the "Splunk errors last 24 hours"-page, and a lot of these guys shows up:

ERROR HttpListener - Exception while processing request from MYIP for /services/receivers/stream?index=main&source=rest&sourcetype=testing: Timeout
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...