Error while Redirect 514 to 9997

[bgaignon]

Hi guys,

I have a source that send log via syslog push tcp 514.
The configuration is working well on my SPlunk test server, I receive the logs.

In production SPlunk is not installed as root so I redirected the port 514 to 9997 like here.

I can see that the iptables has been changed:

 iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 2465 packets, 149K bytes)
 pkts bytes target     prot opt in     out   source         destination
80194 4813K REDIRECT   tcp  --  *      *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  *      *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  lo0    *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  lo0    *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997

But I can't receive my logs and in splunkd.log I receive a lot of messages like:

04-02-2014 10:10:23.776 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44561
04-02-2014 10:10:24.457 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44567

Any Ideas is welcome.

[bgaignon]

Thank you guys.
So yes separate tcp and splunktcp fix the problem.

[sympatiko]

I'm having the same problem. How did you separate the splunktcp to tcp? Thanks

[Ayn]

No, you can't do it like that.

splunktcp is a proprietary protocol used ONLY for forwarding traffic between Splunk instances. Syslog on the other hand is a "raw" and completely different protocol. When you try to send syslog to a port expecting splunktcp traffic, it will just discard the data as it doesn't find it valid.

If you're able to listen on port 514 I'd keep that, and use a raw tcp input there instead.

[lukejadamec]

I was gonna say that, and add that you can configure Splunk to listen for TCP on any port that is not already in use. If your production network blocks 514 then pick another port greater than 1024 that is not already in use on your network.

[sympatiko]

How can I do that. I'm having the same issue. You're help is very much appreciated.

[martin_mueller · Accepted solution]

The splunktcp stanza is for cooked data from Splunk forwarders, don't change that. Instead, add a [tcp://5140] stanza (or any unused port) and redirect 514 there instead of to 9997.

[bgaignon]

I'm listening the port 9997 thanks that: [splunktcp://9997]
Should I add also: [tcp://9997]