Why am I getting error "Received unexpected 369295360 byte message!" connecting to TCP input using java client SDK on Splunk 6.2?

[preben12]

I'm testing the new Splunk version using the java client sdk, and are seeing problems connecting to a tcp receiver port 9997.

When connecting i'm using a serviceargs map with the values {host=localhost, port=9997, scheme=https, username=admin, password=password}

When doing Service.connect(args) i'm getting an error

could not connect to Splunk Server @ localhost:9997 - java.lang.RuntimeException: Remote host closed connection during handshake

From the splunkd.log there are log entries ->

10-28-2014 19:03:49.815 +0100 ERROR TcpInputProc - Message rejected. Received unexpected 369295360 byte message! from src=127.0.0.1:55247. Maximum message allowed: 67108864. (::)

There is no problem when using earlier versions of Splunk eg. 5.3

Best,
Preben

[ineeman · Accepted solution]

The Service.connect method is used to connect to the Splunk REST API, which is against the management port. The port supplied in the ServiceArgs map refers to the management port. Note that TCP ports as configured in "Settings » Forwarding and receiving » Receive data » add new" (as you noted) are not HTTP/HTTPS - they are purely just TCP sockets, with no protocol on them.

I'm not sure how or why it worked in a previous version of Splunk - could it be that 9997 was actually your management port and not one defined as a TCP receiver?

[preben12]

Ahh. you are right. Just took a step deeper. The connection is actually done using http against the management port using the Service.connect. Later the service instance can be used to get a tcp socket connection using the open method ->

 public Socket open(int port) throws IOException {
        return new Socket(this.host, port);
    }

Just forgot

[ineeman]

When you say a "tcp receiver port of 9997" - is this a TCP input port (from inputs.conf) or the management port for Splunk (i.e. the default being 8089)?

[preben12]

It's a tcp receiver port opened using the console -> Settings » Forwarding and receiving » Receive data » add new.

[ineeman]

OK - Service.connect is used to connect to the management port (the REST API) - so you should use that port. Let me know if that makes sense.

[preben12]

I don't think thats correct. Look at the Service class this is a general class for connecting to Splunk using http(s) at the port you configured in the args map.
This has been working fine on previous versions of Splunk.

[ineeman]

Can you share the code you have that you are using? I want to make sure I am following what you are doing.

[preben12]

sure
https://github.com/apache/camel/blob/master/components/camel-splunk/src/main/java/org/apache/camel/c...