Getting Data In

TCP Port Reserved for RAW input

tb5821
Communicator

Hi - I'm trying to have rsyslog send some data on port 4516 to my splunk server running on Centos. I setup a new data input within splunk on this server but I'm seeing the below in the logs.

06-11-2019 19:56:35.508 +0000 INFO  TcpInputProc - removeUnusedAccptors - IPv4 port 4516 not used any more, will clean up
06-11-2019 19:56:35.508 +0000 INFO  TcpInputProc - Closing raw IPv4 port 4516
06-11-2019 19:56:39.105 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-11-2019 19:56:39.106 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-11-2019 19:56:39.106 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 4516 with Non-SSL

What could the issue be? I do see the server listening on that port so I'm not sure its a FW issue

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8191            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:8065          0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN      - 
0 Karma
1 Solution

tb5821
Communicator

Ended up being a missing FW rule ugh - fixed now and ingesting fine.

View solution in original post

0 Karma

tb5821
Communicator

Ended up being a missing FW rule ugh - fixed now and ingesting fine.

0 Karma

DavidHourani
Super Champion

Hi @tb5821,

Are you using an LTM ? It seems there issue is there, have a look here :
https://answers.splunk.com/answers/469248/why-is-tcp-data-not-being-indexed.html

If that's not the case, make sure your syslog source is sending on the defined port and that you're able to telnet from the source on that port.

Cheers,
David

0 Karma

DavidHourani
Super Champion

Hi @tb5821, any updates on this issue ? Is it working now ?

0 Karma

tb5821
Communicator

still trying to figure this one out - confirmed it doesn't look like a FW issue - still seeing port for RAW data in the logs too

06-19-2019 14:24:52.299 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-19-2019 14:24:52.300 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-19-2019 14:24:55.812 +0000 INFO  TcpInputProc - removeUnusedAccptors - IPv4 port 4516 not used any more, will clean up
06-19-2019 14:24:55.812 +0000 INFO  TcpInputProc - Closing raw IPv4 port 4516
06-19-2019 14:24:59.109 +0000 INFO  TcpInputConfig - IPv4 port 4516 is reserved for raw input
06-19-2019 14:24:59.109 +0000 INFO  TcpInputConfig - IPv4 port 4516 will negotiate s2s protocol level 4
06-19-2019 14:24:59.109 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 4516 with Non-SSL
0 Karma

tb5821
Communicator

I still can't telnet from the source to the dest on that port - so I wonder if its something wonky with how splunk "listens" even though netstat says its listening

tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN    
0 Karma

FrankVl
Ultra Champion

But the telnet from localhost worked. right?

Did you run any network capture yet? Does the TCP connect attempt show in the outgoing traffic on the source machine? Does it show on the incoming traffic on the splunk machine?

0 Karma

tb5821
Communicator

Thanks guys.

  • telent works from localhost on that port
  • tcp dump doesn't show network traffic on the dest.
0 Karma

FrankVl
Ultra Champion

if tcpdump is not showing traffic then the tcp attempt is not even reaching your box (assuming you did the capture correctly). So no point in looking at Splunk at this stage, this is clearly something on the source device or in the network in between that is blocking it.

0 Karma

DavidHourani
Super Champion

agree with @FranckVI, also try playing around with the port number see if it stays blocked and if it still gives the same message in the logs.

0 Karma

DavidHourani
Super Champion

@tb5821 could you please try to change the port number, also please share the configuration for the input port.

0 Karma

FrankVl
Ultra Champion

Can you do a netstat -nap, to also show the process name? To ensure it is splunk that is listening on that port.

Not sure what those log messages mean exactly, but they are INFO not error or warnings, so not sure if there is any issue.

Is that syslog sender remote, or on the same host? Have you ran a tcpdump to see if there is any traffic?

0 Karma

tb5821
Communicator
tcp        0      0 0.0.0.0:4516            0.0.0.0:*               LISTEN      19705/splunkd       

output of netstat -nap looks good - the syslog sender is remote.

I'll check out tcpdump

0 Karma

tb5821
Communicator

Looks like tenet from the remote host to splunk on the port above is getting connection refused ...

0 Karma

FrankVl
Ultra Champion

Then my first bet would be a firewall issue. Does the telnet work when you do it locally on the splunk server (to confirm Splunk actually accepts connections)?

0 Karma

tb5821
Communicator

yep telnet to localhost from the splunk host works .... I'll go look at FW stuff

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...