topic Re: TCP Port Reserved for RAW input in Getting Data In

TCP Port Reserved for RAW input

tb5821 — Tue, 11 Jun 2019 20:42:15 GMT

Hi - I'm trying to have rsyslog send some data on port 4516 to my splunk server running on Centos. I setup a new data input within splunk on this server but I'm seeing the below in the logs.

06-11-2019 19:56:35.508 +0000 INFO  TcpInputProc - removeUnusedAccptors - IPv4 port 4516 not used any more, will clean up
06-11-2019 19:56:35.508 +0000 INFO  TcpInputProc - Closing raw IPv4 port 4516
06-11-2019 19:56:39.105 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-11-2019 19:56:39.106 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-11-2019 19:56:39.106 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 4516 with Non-SSL

What could the issue be? I do see the server listening on that port so I'm not sure its a FW issue

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8191            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:8065          0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN      -

Re: TCP Port Reserved for RAW input

FrankVl — Wed, 12 Jun 2019 09:59:16 GMT

Can you do a netstat -nap, to also show the process name? To ensure it is splunk that is listening on that port.

Not sure what those log messages mean exactly, but they are INFO not error or warnings, so not sure if there is any issue.

Is that syslog sender remote, or on the same host? Have you ran a tcpdump to see if there is any traffic?

Re: TCP Port Reserved for RAW input

tb5821 — Wed, 12 Jun 2019 15:31:03 GMT

tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN      19705/splunkd

output of netstat -nap looks good - the syslog sender is remote.

I'll check out tcpdump

Re: TCP Port Reserved for RAW input

tb5821 — Wed, 12 Jun 2019 17:05:17 GMT

Looks like tenet from the remote host to splunk on the port above is getting connection refused ...

Re: TCP Port Reserved for RAW input

FrankVl — Wed, 12 Jun 2019 17:19:07 GMT

Then my first bet would be a firewall issue. Does the telnet work when you do it locally on the splunk server (to confirm Splunk actually accepts connections)?

Re: TCP Port Reserved for RAW input

tb5821 — Wed, 12 Jun 2019 17:27:35 GMT

yep telnet to localhost from the splunk host works .... I'll go look at FW stuff

Re: TCP Port Reserved for RAW input

DavidHourani — Thu, 13 Jun 2019 06:43:40 GMT

Hi @tb5821,

Are you using an LTM ? It seems there issue is there, have a look here :
https://answers.splunk.com/answers/469248/why-is-tcp-data-not-being-indexed.html

If that's not the case, make sure your syslog source is sending on the defined port and that you're able to telnet from the source on that port.

Cheers,
David

Re: TCP Port Reserved for RAW input

DavidHourani — Tue, 18 Jun 2019 08:57:56 GMT

Hi @tb5821, any updates on this issue ? Is it working now ?

Re: TCP Port Reserved for RAW input

tb5821 — Wed, 19 Jun 2019 14:56:33 GMT

still trying to figure this one out - confirmed it doesn't look like a FW issue - still seeing port for RAW data in the logs too

06-19-2019 14:24:52.299 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-19-2019 14:24:52.300 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-19-2019 14:24:55.812 +0000 INFO  TcpInputProc - removeUnusedAccptors - IPv4 port 4516 not used any more, will clean up
06-19-2019 14:24:55.812 +0000 INFO  TcpInputProc - Closing raw IPv4 port 4516
06-19-2019 14:24:59.109 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-19-2019 14:24:59.109 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-19-2019 14:24:59.109 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 4516 with Non-SSL

Re: TCP Port Reserved for RAW input

tb5821 — Wed, 19 Jun 2019 14:57:51 GMT

I still can't telnet from the source to the dest on that port - so I wonder if its something wonky with how splunk "listens" even though netstat says its listening

tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN

Re: TCP Port Reserved for RAW input

FrankVl — Wed, 19 Jun 2019 15:04:49 GMT

But the telnet from localhost worked. right?

Did you run any network capture yet? Does the TCP connect attempt show in the outgoing traffic on the source machine? Does it show on the incoming traffic on the splunk machine?

Re: TCP Port Reserved for RAW input

DavidHourani — Wed, 19 Jun 2019 15:28:24 GMT

@tb5821 could you please try to change the port number, also please share the configuration for the input port.

Re: TCP Port Reserved for RAW input

tb5821 — Wed, 19 Jun 2019 15:49:39 GMT

Thanks guys.

telent works from localhost on that port
tcp dump doesn't show network traffic on the dest.

Re: TCP Port Reserved for RAW input

FrankVl — Wed, 19 Jun 2019 15:53:25 GMT

if tcpdump is not showing traffic then the tcp attempt is not even reaching your box (assuming you did the capture correctly). So no point in looking at Splunk at this stage, this is clearly something on the source device or in the network in between that is blocking it.

Re: TCP Port Reserved for RAW input

DavidHourani — Wed, 19 Jun 2019 16:10:09 GMT

agree with @FranckVI, also try playing around with the port number see if it stays blocked and if it still gives the same message in the logs.

Re: TCP Port Reserved for RAW input

tb5821 — Thu, 20 Jun 2019 14:12:08 GMT

Ended up being a missing FW rule ugh - fixed now and ingesting fine.