topic Re: Error while Redirect 514 to 9997 in Getting Data In

Error while Redirect 514 to 9997

bgaignon — Wed, 02 Apr 2014 14:29:36 GMT

Hi guys,

I have a source that send log via syslog push tcp 514.
The configuration is working well on my SPlunk test server, I receive the logs.

In production SPlunk is not installed as root so I redirected the port 514 to 9997 like here.

I can see that the iptables has been changed:

 iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 2465 packets, 149K bytes)
 pkts bytes target     prot opt in     out   source         destination
80194 4813K REDIRECT   tcp  --  *      *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  *      *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  lo0    *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  lo0    *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997

But I can't receive my logs and in splunkd.log I receive a lot of messages like:

04-02-2014 10:10:23.776 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44561
04-02-2014 10:10:24.457 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44567

Any Ideas is welcome.

Re: Error while Redirect 514 to 9997

bgaignon — Wed, 02 Apr 2014 14:44:10 GMT

I'm listening the port 9997 thanks that: [splunktcp://9997]
Should I add also: [tcp://9997]

Re: Error while Redirect 514 to 9997

martin_mueller — Wed, 02 Apr 2014 16:12:42 GMT

The splunktcp stanza is for cooked data from Splunk forwarders, don't change that. Instead, add a [tcp://5140] stanza (or any unused port) and redirect 514 there instead of to 9997.

Re: Error while Redirect 514 to 9997

Ayn — Wed, 02 Apr 2014 16:16:30 GMT

No, you can't do it like that.

splunktcp is a proprietary protocol used ONLY for forwarding traffic between Splunk instances. Syslog on the other hand is a "raw" and completely different protocol. When you try to send syslog to a port expecting splunktcp traffic, it will just discard the data as it doesn't find it valid.

If you're able to listen on port 514 I'd keep that, and use a raw tcp input there instead.

Re: Error while Redirect 514 to 9997

lukejadamec — Wed, 02 Apr 2014 16:50:43 GMT

I was gonna say that, and add that you can configure Splunk to listen for TCP on any port that is not already in use. If your production network blocks 514 then pick another port greater than 1024 that is not already in use on your network.

Re: Error while Redirect 514 to 9997

bgaignon — Wed, 02 Apr 2014 17:06:47 GMT

Thank you guys.
So yes separate tcp and splunktcp fix the problem.

Re: Error while Redirect 514 to 9997

sympatiko — Wed, 19 Nov 2014 00:48:22 GMT

I'm having the same problem. How did you separate the splunktcp to tcp? Thanks

Re: Error while Redirect 514 to 9997

sympatiko — Wed, 19 Nov 2014 01:19:21 GMT

How can I do that. I'm having the same issue. You're help is very much appreciated.