Archive

Error while Redirect 514 to 9997

Path Finder

Hi guys,

I have a source that send log via syslog push tcp 514.
The configuration is working well on my SPlunk test server, I receive the logs.

In production SPlunk is not installed as root so I redirected the port 514 to 9997 like here.

I can see that the iptables has been changed:

 iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 2465 packets, 149K bytes)
 pkts bytes target     prot opt in     out   source         destination
80194 4813K REDIRECT   tcp  --  *      *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  *      *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  lo0    *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  lo0    *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997

But I can't receive my logs and in splunkd.log I receive a lot of messages like:

04-02-2014 10:10:23.776 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44561
04-02-2014 10:10:24.457 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44567

Any Ideas is welcome.

0 Karma
1 Solution

SplunkTrust
SplunkTrust

The splunktcp stanza is for cooked data from Splunk forwarders, don't change that. Instead, add a [tcp://5140] stanza (or any unused port) and redirect 514 there instead of to 9997.

View solution in original post

0 Karma

Path Finder

Thank you guys.
So yes separate tcp and splunktcp fix the problem.

0 Karma

Communicator

I'm having the same problem. How did you separate the splunktcp to tcp? Thanks

0 Karma

Legend

No, you can't do it like that.

splunktcp is a proprietary protocol used ONLY for forwarding traffic between Splunk instances. Syslog on the other hand is a "raw" and completely different protocol. When you try to send syslog to a port expecting splunktcp traffic, it will just discard the data as it doesn't find it valid.

If you're able to listen on port 514 I'd keep that, and use a raw tcp input there instead.

Super Champion

I was gonna say that, and add that you can configure Splunk to listen for TCP on any port that is not already in use. If your production network blocks 514 then pick another port greater than 1024 that is not already in use on your network.

0 Karma

Communicator

How can I do that. I'm having the same issue. You're help is very much appreciated.

0 Karma

SplunkTrust
SplunkTrust

The splunktcp stanza is for cooked data from Splunk forwarders, don't change that. Instead, add a [tcp://5140] stanza (or any unused port) and redirect 514 there instead of to 9997.

View solution in original post

0 Karma

Path Finder

I'm listening the port 9997 thanks that: [splunktcp://9997]
Should I add also: [tcp://9997]

0 Karma