Hi guys,
I have a source that send log via syslog push tcp 514.
The configuration is working well on my SPlunk test server, I receive the logs.
In production SPlunk is not installed as root so I redirected the port 514 to 9997 like here.
I can see that the iptables has been changed:
iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 2465 packets, 149K bytes)
pkts bytes target prot opt in out source destination
80194 4813K REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:514 redir ports 9997
0 0 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:514 redir ports 9997
0 0 REDIRECT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:514 redir ports 9997
0 0 REDIRECT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 redir ports 9997
0 0 REDIRECT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 redir ports 9997
0 0 REDIRECT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:514 redir ports 9997
0 0 REDIRECT tcp -- lo0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:514 redir ports 9997
0 0 REDIRECT udp -- lo0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 redir ports 9997
But I can't receive my logs and in splunkd.log I receive a lot of messages like:
04-02-2014 10:10:23.776 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44561
04-02-2014 10:10:24.457 -0400 ERROR TcpInputProc - Received unexpected 1009857598 byte message (Invalid payload_size=1009857598 received while in parseState=1)! from src=100.101.102.103:44567
Any Ideas is welcome.
The splunktcp stanza is for cooked data from Splunk forwarders, don't change that. Instead, add a [tcp://5140]
stanza (or any unused port) and redirect 514 there instead of to 9997.
Thank you guys.
So yes separate tcp and splunktcp fix the problem.
I'm having the same problem. How did you separate the splunktcp to tcp? Thanks
No, you can't do it like that.
splunktcp is a proprietary protocol used ONLY for forwarding traffic between Splunk instances. Syslog on the other hand is a "raw" and completely different protocol. When you try to send syslog to a port expecting splunktcp traffic, it will just discard the data as it doesn't find it valid.
If you're able to listen on port 514 I'd keep that, and use a raw tcp input there instead.
I was gonna say that, and add that you can configure Splunk to listen for TCP on any port that is not already in use. If your production network blocks 514 then pick another port greater than 1024 that is not already in use on your network.
How can I do that. I'm having the same issue. You're help is very much appreciated.
The splunktcp stanza is for cooked data from Splunk forwarders, don't change that. Instead, add a [tcp://5140]
stanza (or any unused port) and redirect 514 there instead of to 9997.
I'm listening the port 9997 thanks that: [splunktcp://9997]
Should I add also: [tcp://9997]