Splunk can't get data from remote machines

[pacifikn]

Dear All,

Greetings!! 

I need your help,

 Splunk server for log collector x.x.x.x port=y  can't receive data from all syslog sender that send data to this log collector server. And I have check the port by doing telnet 

-->   telnet IP port  :     telnet x.x.x.x  y and It is not responding,

what to do/check if you find this IP and port is not responding.....and this cause to not receive logs from all syslog sender.  

Kindly help me how to troubleshoot this, Thank you in advance

[nickhills]

If the syslog receiver is using UDP you can't test it with telnet, instead try netcat.

nc -z -v -u <your_IP> <your_port_number>

Although, a good start is to use netstat on the recieving host and confirm the host is listening on the right interface/port/proto!

netstat -ln|grep <your_port_number>

[pacifikn]

Dear nickhills,

I have tried the the below, here is what i get:

1.         nc -z -v -u  public-IP  port 

output:

Ncat: Version 7.50 (https:/nmap.org/ncat)

Ncat: Connected to Public-IP:port.

Ncat: UDP Packet sent successfully

Ncat: 1 bytes sent, 0 bytes received in 2.06 seconds.

2.      netstat -ln | grep port 

tcp    0     0.0.0.0:port       0.0.0.0:*            LISTEN

udp   0      0.0.0.0:port       0.0.0.0:*                         

That are the output of the above command, in the second command there's no listening? how can I fix this? what is the issue ?

[richgalloway]

Have you verified something is listening to that address and port?  Have you checked your firewalls?

[pacifikn]

dear @richgalloway 

May you help me and share with me the command I can use to check the below information you shared? I use centos 7 ? 

I htave checked the syslog sender all are configured well to send logs into Splunk server log collector, But the problem is that I can't receive logs into splunk server log collector , what are the all troubleshooting to go through to check the root cause??kindly share with me all the command i can use to check all the services? 

Thank you

[richgalloway]

To determine if a process is listening to a port, use the netstat command.

netstat -ln | grep y

Since I don't know what firewall you use I can't suggest commands to check it.  Consult your system admin (or Google). 

[pacifikn]

@richgalloway 

netstat -ln | grep 514

tcp      0       0.0.0.0:514         0.0.0.0:*     LISTEN

udp     0      0.0.0.0:514          0.0.0.0:*                        

this the output I got from the above command.  In udp line there's no LISTEN info appeared , this is how should be or this means that is not listening? 

AND

nc -u x.x.x.x 514 , I got flashing point

[richgalloway · Accepted solution]

You should ask your network team for help with the ASA.

The server on which Splunk is running may have its own software firewall (iptables or the like).  It, too, must be checked to make sure it's not blocking the port.