I have a scenario here
The data is being onboarded from one particular set of forwarders with ip 172.30.xx.xxx and the other set with 172.16.xx.xxx is not being onboarded. I checked the config files and also checked the ports. But i am unable to send the data . My team says that data is blocked with particular 172.16.xx.xxx set of forwarders. Our plan is use to implement using syslog server along with Heavy forwarders. But my doubt is if the servers have no connectivity how will it help. Please help me guide through more troubleshooting steps as before we go with syslog i wanted to know the right reason for data not being onboarded. Thanks in advance
This is dependent of your infrastructure, still you can try following.
netstat -anp | grep 9997
and on windows netstat -an
it should show port 9997 listening, in linux it will also show process name.tcpdump -A -i <ethernet card> port 9997
in windows you can use tcpview utility available to download in windows sysinternals.There are many more tests I can suggest, please respond with the results of the above.
Also you mentioned My team says that data is blocked with particular 172.16.xx.xxx set of forwarders.
What does data is blocked means? is it blocked at network level or is it just not seen in the Indexer?
The reason they r blocking is becoz they are internal servers which cannot communicate with splunk cloud . So the plan is to implement using syslog server along with HF to communicate with splunk ..do you guys prefer going through syslog and HF or Just with HF would be good
In such a scenario, I will always prefer setting up a syslog server and then have a UF / HF on the syslog server. This will be a better solution than having only HF. The Syslog server helps manage the log storage in a better way, and Splunk HF will give you a capability to control the data pipeline.
Incase if network connectivity faces issue, HF can prevent data loss. It can also wait for the network connectivity resume to share next copy of data.
Do you have the process or config to be done . UF-Syslogserver-HF-SPlunkcloud
I can do the complete configuration for the above scenario. But it will be too big to explain here, also it is complex. I will just write in brief about this here.
Can I have the configuration for syslog server like the Disk space, CPU and RAM size.
All the Compute and Storage calculations are completely dependent on your infrastructure and the amount of data you are ingesting per day. It also has multiple aspects such as the retention of logs on the Syslog server. There is not a 'standard' set for how much compute and storage is required for a Syslog server.
You need to find some help in-house, your Infrastructure team can help you with this.
Thank you . The telnet 9997 says connection refused . It connects when i restart my HF and after like around 5 minutes it says connection refused.Please help and thanks in advance
I consider the servers can communicate and are able to ping each other.
Can you see the port 9997 as LISTENING when you run netstat -an on the HF?
If yes, then the Linux Firewalld / IPtables / SeLinux is not allowing the port. Please allow it in the firewall. If you do not know, please take help of IT Team.
Also there is a possibility that Network Firewall Coming between HF and Syslog might be not allowing the port.
Right now I am directly trying to send to HF as I want to ultimately check if it reaches the splunk cloud. When I hit netsta on HF it says CLOSE_WAIT and ESTABLISHED . here i am reciving it through 9998 and sending to cloud vis 9997.
It should be LISTENING or ESTABLISHED on the HF for port 9998. If it is listening, then you can Telnet from the server to the HF on port 9998. If the telnet is successfull, there is no connectivity problem. If the telnet shows access denied, then there is a some filtering happening either through a network firewall or Linux firewall.
If this is a linux machine, you can run tcpdump to see if you are receiving data.
tcpdump -A -i <ethernet card name> port 9998
It should show packets flowing.
If you think my answer is helpful, please upvote and mark my answer as solution. Thanks.
This is dependent of your infrastructure, still you can try following.
netstat -anp | grep 9997
and on windows netstat -an
it should show port 9997 listening, in linux it will also show process name.tcpdump -A -i <ethernet card> port 9997
in windows you can use tcpview utility available to download in windows sysinternals.There are many more tests I can suggest, please respond with the results of the above.
Also you mentioned My team says that data is blocked with particular 172.16.xx.xxx set of forwarders.
What does data is blocked means? is it blocked at network level or is it just not seen in the Indexer?
Thank you for the troubleshooting steps . The reason they r blocking is becoz they are internal servers which cannot communicate with splunk cloud . So the plan is to implement using syslog server along with HF to communicate with splunk ..do you guys prefer going through syslog and HF or Just with HF would be good
Check if the broken forwarders have the correct outputs.conf
to send to the Indexers. Check if you can telnet to an indexer on port 9997. You will not be able to login but you should get a login prompt; if you get a timeout then either you don't have a route to the indexer or your firewall is blocking the traffic.
i dont have indexer ip just the url. will try to get ip of indexer . but using netstat -an |grep 9997 i could not find any process running that means the forwarding port 9997 is not open right ?.
The official trouble shooting guide for this case is at I can't find my data!
Have you checked the firewalls? What do the logs on the 172.16 forwarders say?
I could not check the firewalls?. One thing i noticed is that when i checked forwarding port 9997 using netstat -an |grep 9997 in 172.30.xx.xxx, I could see its running but on 172.16.xx.xxx I dont see any process running also seems like the port is not open. Could this be the problem ?. Also regarding firewall check , I need to figure out with network /operations team may be?
You may not be able to check the firewalls, but that doesn't mean they're not the problem. Check with your network team about the network firewalls and you system admins for the server firewalls.
Verify the 172.16 forwarders are running.