Getting Data In

TCP Port Reserved for RAW input

Communicator

Hi - I'm trying to have rsyslog send some data on port 4516 to my splunk server running on Centos. I setup a new data input within splunk on this server but I'm seeing the below in the logs.

06-11-2019 19:56:35.508 +0000 INFO  TcpInputProc - removeUnusedAccptors - IPv4 port 4516 not used any more, will clean up
06-11-2019 19:56:35.508 +0000 INFO  TcpInputProc - Closing raw IPv4 port 4516
06-11-2019 19:56:39.105 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-11-2019 19:56:39.106 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-11-2019 19:56:39.106 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 4516 with Non-SSL

What could the issue be? I do see the server listening on that port so I'm not sure its a FW issue

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8191            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:8065          0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN      - 
0 Karma
1 Solution

Communicator

Ended up being a missing FW rule ugh - fixed now and ingesting fine.

View solution in original post

0 Karma

Communicator

Ended up being a missing FW rule ugh - fixed now and ingesting fine.

View solution in original post

0 Karma

Super Champion

Hi @tb5821,

Are you using an LTM ? It seems there issue is there, have a look here :
https://answers.splunk.com/answers/469248/why-is-tcp-data-not-being-indexed.html

If that's not the case, make sure your syslog source is sending on the defined port and that you're able to telnet from the source on that port.

Cheers,
David

0 Karma

Super Champion

Hi @tb5821, any updates on this issue ? Is it working now ?

0 Karma

Communicator

still trying to figure this one out - confirmed it doesn't look like a FW issue - still seeing port for RAW data in the logs too

06-19-2019 14:24:52.299 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-19-2019 14:24:52.300 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-19-2019 14:24:55.812 +0000 INFO  TcpInputProc - removeUnusedAccptors - IPv4 port 4516 not used any more, will clean up
06-19-2019 14:24:55.812 +0000 INFO  TcpInputProc - Closing raw IPv4 port 4516
06-19-2019 14:24:59.109 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-19-2019 14:24:59.109 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-19-2019 14:24:59.109 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 4516 with Non-SSL
0 Karma

Communicator

I still can't telnet from the source to the dest on that port - so I wonder if its something wonky with how splunk "listens" even though netstat says its listening

tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN    
0 Karma

Ultra Champion

But the telnet from localhost worked. right?

Did you run any network capture yet? Does the TCP connect attempt show in the outgoing traffic on the source machine? Does it show on the incoming traffic on the splunk machine?

0 Karma

Communicator

Thanks guys.

  • telent works from localhost on that port
  • tcp dump doesn't show network traffic on the dest.
0 Karma

Ultra Champion

if tcpdump is not showing traffic then the tcp attempt is not even reaching your box (assuming you did the capture correctly). So no point in looking at Splunk at this stage, this is clearly something on the source device or in the network in between that is blocking it.

0 Karma

Super Champion

agree with @FranckVI, also try playing around with the port number see if it stays blocked and if it still gives the same message in the logs.

0 Karma

Super Champion

@tb5821 could you please try to change the port number, also please share the configuration for the input port.

0 Karma

Ultra Champion

Can you do a netstat -nap, to also show the process name? To ensure it is splunk that is listening on that port.

Not sure what those log messages mean exactly, but they are INFO not error or warnings, so not sure if there is any issue.

Is that syslog sender remote, or on the same host? Have you ran a tcpdump to see if there is any traffic?

0 Karma

Communicator
tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN      19705/splunkd       

output of netstat -nap looks good - the syslog sender is remote.

I'll check out tcpdump

0 Karma

Communicator

Looks like tenet from the remote host to splunk on the port above is getting connection refused ...

0 Karma

Ultra Champion

Then my first bet would be a firewall issue. Does the telnet work when you do it locally on the splunk server (to confirm Splunk actually accepts connections)?

0 Karma

Communicator

yep telnet to localhost from the splunk host works .... I'll go look at FW stuff

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!