Getting Data In

onboarding issues

bobba40
New Member

I have a scenario here

The data is being onboarded from one particular set of forwarders with ip 172.30.xx.xxx and the other set with 172.16.xx.xxx is not being onboarded. I checked the config files and also checked the ports. But i am unable to send the data . My team says that data is blocked with particular 172.16.xx.xxx set of forwarders. Our plan is use to implement using syslog server along with Heavy forwarders. But my doubt is if the servers have no connectivity how will it help. Please help me guide through more troubleshooting steps as before we go with syslog i wanted to know the right reason for data not being onboarded. Thanks in advance

0 Karma
1 Solution

ashutoshab
Communicator

This is dependent of your infrastructure, still you can try following.

  1. Check you have the correct configurations in outputs.conf. i.e. the IP address of indexer is correct with the receiving port default to 9997.
  2. Try to Telnet from UF to Indexer, if the telnet is succesfull there is no connectivity issue.
  3. See if the Firewall / iptables / selinux / firewalld / ufw is not allowing any service or port on the UF as well as on Indexer Side.
  4. See if the Route is present between network of UF and network where Indexer is present.
  5. See If any network firewall blocks traffic in between UF and Indexers.
  6. See if you Opened the receiving port on indexer correctly.
  7. On the indexer, assuming you are receiving on default port i.e. 9997. try to do netstat i.e Linux netstat -anp | grep 9997 and on windows netstat -an it should show port 9997 listening, in linux it will also show process name.
  8. If you find ports are listening as in step 7, you can try seeing the packets such as on Linux tcpdump -A -i <ethernet card> port 9997 in windows you can use tcpview utility available to download in windows sysinternals.

There are many more tests I can suggest, please respond with the results of the above.

Also you mentioned My team says that data is blocked with particular 172.16.xx.xxx set of forwarders. What does data is blocked means? is it blocked at network level or is it just not seen in the Indexer?

View solution in original post

0 Karma

bobba40
New Member

The reason they r blocking is becoz they are internal servers which cannot communicate with splunk cloud . So the plan is to implement using syslog server along with HF to communicate with splunk ..do you guys prefer going through syslog and HF or Just with HF would be good

0 Karma

ashutoshab
Communicator

In such a scenario, I will always prefer setting up a syslog server and then have a UF / HF on the syslog server. This will be a better solution than having only HF. The Syslog server helps manage the log storage in a better way, and Splunk HF will give you a capability to control the data pipeline.
Incase if network connectivity faces issue, HF can prevent data loss. It can also wait for the network connectivity resume to share next copy of data.

0 Karma

bobba40
New Member

Do you have the process or config to be done . UF-Syslogserver-HF-SPlunkcloud

0 Karma

ashutoshab
Communicator

I can do the complete configuration for the above scenario. But it will be too big to explain here, also it is complex. I will just write in brief about this here.

  1. First, create your Syslog Server. You can utilize capabilities of Linux based Syslog daemons such as Rsyslog or Syslog-ng.
  2. Install HF on the Syslog Server and configure it to forward data to Splunk Cloud.
  3. Configure Splunk Cloud to receive the data.
  4. Install and Configure the UFs on end machines. Here you can benefit by using a Deployment Server for managing the configurations across the servers.
0 Karma

bobba40
New Member

Can I have the configuration for syslog server like the Disk space, CPU and RAM size.

0 Karma

ashutoshab
Communicator

All the Compute and Storage calculations are completely dependent on your infrastructure and the amount of data you are ingesting per day. It also has multiple aspects such as the retention of logs on the Syslog server. There is not a 'standard' set for how much compute and storage is required for a Syslog server.

You need to find some help in-house, your Infrastructure team can help you with this.

0 Karma

bobba40
New Member

Thank you . The telnet 9997 says connection refused . It connects when i restart my HF and after like around 5 minutes it says connection refused.Please help and thanks in advance

0 Karma

ashutoshab
Communicator

I consider the servers can communicate and are able to ping each other.

Can you see the port 9997 as LISTENING when you run netstat -an on the HF?

If yes, then the Linux Firewalld / IPtables / SeLinux is not allowing the port. Please allow it in the firewall. If you do not know, please take help of IT Team.
Also there is a possibility that Network Firewall Coming between HF and Syslog might be not allowing the port.

0 Karma

bobba40
New Member

Right now I am directly trying to send to HF as I want to ultimately check if it reaches the splunk cloud. When I hit netsta on HF it says CLOSE_WAIT and ESTABLISHED . here i am reciving it through 9998 and sending to cloud vis 9997.

0 Karma

ashutoshab
Communicator

It should be LISTENING or ESTABLISHED on the HF for port 9998. If it is listening, then you can Telnet from the server to the HF on port 9998. If the telnet is successfull, there is no connectivity problem. If the telnet shows access denied, then there is a some filtering happening either through a network firewall or Linux firewall.

If this is a linux machine, you can run tcpdump to see if you are receiving data.

tcpdump -A -i <ethernet card name> port 9998

It should show packets flowing.

0 Karma

ashutoshab
Communicator

If you think my answer is helpful, please upvote and mark my answer as solution. Thanks.

0 Karma

ashutoshab
Communicator

This is dependent of your infrastructure, still you can try following.

  1. Check you have the correct configurations in outputs.conf. i.e. the IP address of indexer is correct with the receiving port default to 9997.
  2. Try to Telnet from UF to Indexer, if the telnet is succesfull there is no connectivity issue.
  3. See if the Firewall / iptables / selinux / firewalld / ufw is not allowing any service or port on the UF as well as on Indexer Side.
  4. See if the Route is present between network of UF and network where Indexer is present.
  5. See If any network firewall blocks traffic in between UF and Indexers.
  6. See if you Opened the receiving port on indexer correctly.
  7. On the indexer, assuming you are receiving on default port i.e. 9997. try to do netstat i.e Linux netstat -anp | grep 9997 and on windows netstat -an it should show port 9997 listening, in linux it will also show process name.
  8. If you find ports are listening as in step 7, you can try seeing the packets such as on Linux tcpdump -A -i <ethernet card> port 9997 in windows you can use tcpview utility available to download in windows sysinternals.

There are many more tests I can suggest, please respond with the results of the above.

Also you mentioned My team says that data is blocked with particular 172.16.xx.xxx set of forwarders. What does data is blocked means? is it blocked at network level or is it just not seen in the Indexer?

View solution in original post

0 Karma

bobba40
New Member

Thank you for the troubleshooting steps . The reason they r blocking is becoz they are internal servers which cannot communicate with splunk cloud . So the plan is to implement using syslog server along with HF to communicate with splunk ..do you guys prefer going through syslog and HF or Just with HF would be good

0 Karma

woodcock
Esteemed Legend

Check if the broken forwarders have the correct outputs.conf to send to the Indexers. Check if you can telnet to an indexer on port 9997. You will not be able to login but you should get a login prompt; if you get a timeout then either you don't have a route to the indexer or your firewall is blocking the traffic.

0 Karma

bobba40
New Member

i dont have indexer ip just the url. will try to get ip of indexer . but using netstat -an |grep 9997 i could not find any process running that means the forwarding port 9997 is not open right ?.

0 Karma

ddrillic
Ultra Champion

The official trouble shooting guide for this case is at I can't find my data!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you checked the firewalls? What do the logs on the 172.16 forwarders say?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

bobba40
New Member

I could not check the firewalls?. One thing i noticed is that when i checked forwarding port 9997 using netstat -an |grep 9997 in 172.30.xx.xxx, I could see its running but on 172.16.xx.xxx I dont see any process running also seems like the port is not open. Could this be the problem ?. Also regarding firewall check , I need to figure out with network /operations team may be?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You may not be able to check the firewalls, but that doesn't mean they're not the problem. Check with your network team about the network firewalls and you system admins for the server firewalls.
Verify the 172.16 forwarders are running.

---
If this reply helps you, an upvote would be appreciated.
0 Karma