TCP Port Reserved for RAW input

[tb5821]

Hi - I'm trying to have rsyslog send some data on port 4516 to my splunk server running on Centos. I setup a new data input within splunk on this server but I'm seeing the below in the logs.

06-11-2019 19:56:35.508 +0000 INFO  TcpInputProc - removeUnusedAccptors - IPv4 port 4516 not used any more, will clean up
06-11-2019 19:56:35.508 +0000 INFO  TcpInputProc - Closing raw IPv4 port 4516
06-11-2019 19:56:39.105 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-11-2019 19:56:39.106 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-11-2019 19:56:39.106 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 4516 with Non-SSL

What could the issue be? I do see the server listening on that port so I'm not sure its a FW issue

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8191            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:8065          0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN      - 

[tb5821 · Accepted solution]

Ended up being a missing FW rule ugh - fixed now and ingesting fine.

[DavidHourani]

Hi @tb5821,

Are you using an LTM ? It seems there issue is there, have a look here :
https://answers.splunk.com/answers/469248/why-is-tcp-data-not-being-indexed.html

If that's not the case, make sure your syslog source is sending on the defined port and that you're able to telnet from the source on that port.

Cheers,
David

[DavidHourani]

Hi @tb5821, any updates on this issue ? Is it working now ?

[tb5821]

still trying to figure this one out - confirmed it doesn't look like a FW issue - still seeing port for RAW data in the logs too

06-19-2019 14:24:52.299 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-19-2019 14:24:52.300 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-19-2019 14:24:55.812 +0000 INFO  TcpInputProc - removeUnusedAccptors - IPv4 port 4516 not used any more, will clean up
06-19-2019 14:24:55.812 +0000 INFO  TcpInputProc - Closing raw IPv4 port 4516
06-19-2019 14:24:59.109 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-19-2019 14:24:59.109 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-19-2019 14:24:59.109 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 4516 with Non-SSL

[tb5821]

I still can't telnet from the source to the dest on that port - so I wonder if its something wonky with how splunk "listens" even though netstat says its listening

tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN    

[FrankVl]

But the telnet from localhost worked. right?

Did you run any network capture yet? Does the TCP connect attempt show in the outgoing traffic on the source machine? Does it show on the incoming traffic on the splunk machine?

[tb5821]

Thanks guys.

• telent works from localhost on that port
• tcp dump doesn't show network traffic on the dest.

[FrankVl]

if tcpdump is not showing traffic then the tcp attempt is not even reaching your box (assuming you did the capture correctly). So no point in looking at Splunk at this stage, this is clearly something on the source device or in the network in between that is blocking it.

[DavidHourani]

agree with @FranckVI, also try playing around with the port number see if it stays blocked and if it still gives the same message in the logs.

[DavidHourani]

@tb5821 could you please try to change the port number, also please share the configuration for the input port.

[FrankVl]

Can you do a netstat -nap, to also show the process name? To ensure it is splunk that is listening on that port.

Not sure what those log messages mean exactly, but they are INFO not error or warnings, so not sure if there is any issue.

Is that syslog sender remote, or on the same host? Have you ran a tcpdump to see if there is any traffic?

[tb5821]

tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN      19705/splunkd       

output of netstat -nap looks good - the syslog sender is remote.

I'll check out tcpdump

[tb5821]

Looks like tenet from the remote host to splunk on the port above is getting connection refused ...

[FrankVl]

Then my first bet would be a firewall issue. Does the telnet work when you do it locally on the splunk server (to confirm Splunk actually accepts connections)?

[tb5821]

yep telnet to localhost from the splunk host works .... I'll go look at FW stuff