topic Re: Splunk can't get data from remote machines in Splunk Enterprise

Splunk can't get data from remote machines

pacifikn — Mon, 21 Dec 2020 14:58:53 GMT

Dear All,

Greetings!!

I need your help,

Splunk server for log collector x.x.x.x port=y can't receive data from all syslog sender that send data to this log collector server. And I have check the port by doing telnet

--> telnet IP port : telnet x.x.x.x y and It is not responding,

what to do/check if you find this IP and port is not responding.....and this cause to not receive logs from all syslog sender.

Kindly help me how to troubleshoot this, Thank you in advance

Re: Splunk can't get data from remote machines

richgalloway — Mon, 21 Dec 2020 15:09:18 GMT

Have you verified something is listening to that address and port? Have you checked your firewalls?

Re: Splunk can't get data from remote machines

nickhills — Mon, 21 Dec 2020 15:28:07 GMT

If the syslog receiver is using UDP you can't test it with telnet, instead try netcat.

nc -z -v -u <your_IP> <your_port_number>

Although, a good start is to use netstat on the recieving host and confirm the host is listening on the right interface/port/proto!

netstat -ln|grep <your_port_number>

Re: Splunk can't get data from remote machines

pacifikn — Tue, 22 Dec 2020 05:23:15 GMT

Dear nickhills,

I have tried the the below, here is what i get:

1. nc -z -v -u public-IP port

output:

Ncat: Version 7.50 (https:/nmap.org/ncat)

Ncat: Connected to Public-IP:port.

Ncat: UDP Packet sent successfully

Ncat: 1 bytes sent, 0 bytes received in 2.06 seconds.

2. netstat -ln | grep port

tcp 0 0.0.0.0:port 0.0.0.0:* LISTEN

udp 0 0.0.0.0:port 0.0.0.0:*

That are the output of the above command, in the second command there's no listening? how can I fix this? what is the issue ?

Re: Splunk can't get data from remote machines

pacifikn — Tue, 22 Dec 2020 05:32:03 GMT

dear @richgalloway

May you help me and share with me the command I can use to check the below information you shared? I use centos 7 ?

I htave checked the syslog sender all are configured well to send logs into Splunk server log collector, But the problem is that I can't receive logs into splunk server log collector , what are the all troubleshooting to go through to check the root cause??kindly share with me all the command i can use to check all the services?

Thank you

Re: Splunk can't get data from remote machines

richgalloway — Tue, 22 Dec 2020 14:05:44 GMT

To determine if a process is listening to a port, use the netstat command.

netstat -ln | grep y

Since I don't know what firewall you use I can't suggest commands to check it. Consult your system admin (or Google).

Re: Splunk can't get data from remote machines

pacifikn — Tue, 22 Dec 2020 14:54:11 GMT

@richgalloway

netstat -ln | grep 514

tcp 0 0.0.0.0:514 0.0.0.0:* LISTEN

udp 0 0.0.0.0:514 0.0.0.0:*

this the output I got from the above command. In udp line there's no LISTEN info appeared , this is how should be or this means that is not listening?

AND

nc -u x.x.x.x 514 , I got flashing point

Re: Splunk can't get data from remote machines

richgalloway — Tue, 22 Dec 2020 14:33:34 GMT

You should ask your network team for help with the ASA.

The server on which Splunk is running may have its own software firewall (iptables or the like). It, too, must be checked to make sure it's not blocking the port.