Getting Data In

Error while Redirect 514 to 9997

bgaignon
Path Finder

Hi guys,

I have a source that send log via syslog push tcp 514.
The configuration is working well on my SPlunk test server, I receive the logs.

In production SPlunk is not installed as root so I redirected the port 514 to 9997 like here.

I can see that the iptables has been changed:

 iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 2465 packets, 149K bytes)
 pkts bytes target     prot opt in     out   source         destination
80194 4813K REDIRECT   tcp  --  *      *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  *      *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  eth0   *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   tcp  --  lo0    *     0.0.0.0/0      0.0.0.0/0     tcp dpt:514 redir ports 9997
    0     0 REDIRECT   udp  --  lo0    *     0.0.0.0/0      0.0.0.0/0     udp dpt:514 redir ports 9997

But I can't receive my logs and in splunkd.log I receive a lot of messages like:

04-02-2014 10:10:23.776 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44561
04-02-2014 10:10:24.457 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44567

Any Ideas is welcome.

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

The splunktcp stanza is for cooked data from Splunk forwarders, don't change that. Instead, add a [tcp://5140] stanza (or any unused port) and redirect 514 there instead of to 9997.

View solution in original post

0 Karma

bgaignon
Path Finder

Thank you guys.
So yes separate tcp and splunktcp fix the problem.

0 Karma

sympatiko
Communicator

I'm having the same problem. How did you separate the splunktcp to tcp? Thanks

0 Karma

Ayn
Legend

No, you can't do it like that.

splunktcp is a proprietary protocol used ONLY for forwarding traffic between Splunk instances. Syslog on the other hand is a "raw" and completely different protocol. When you try to send syslog to a port expecting splunktcp traffic, it will just discard the data as it doesn't find it valid.

If you're able to listen on port 514 I'd keep that, and use a raw tcp input there instead.

lukejadamec
Super Champion

I was gonna say that, and add that you can configure Splunk to listen for TCP on any port that is not already in use. If your production network blocks 514 then pick another port greater than 1024 that is not already in use on your network.

0 Karma

sympatiko
Communicator

How can I do that. I'm having the same issue. You're help is very much appreciated.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The splunktcp stanza is for cooked data from Splunk forwarders, don't change that. Instead, add a [tcp://5140] stanza (or any unused port) and redirect 514 there instead of to 9997.

0 Karma

bgaignon
Path Finder

I'm listening the port 9997 thanks that: [splunktcp://9997]
Should I add also: [tcp://9997]

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...