Two of my servers not reporting in Splunk. They are running in windows server 2012 r2 std and 2016 datacenter. Splunk universal forwarder 7.2.0 installed in both servers.
Please find find my below observations:
1.Iam able to telnet the below IPs.
telnet 54.157.x.x 9997
telnet 34.197.x.x 9997
telnet 35.175.x.x 9997
telnet 54.241.x.x 443
2.So port is allowed but when i run netstat -a 9997 port not shows.
3.Splunk service is running in both servers(But when i try to restart, first time it shows error (windows cannot stop splunk forwarder service on local computer Error:1053) and the service gets stopped but iam able to start the service anyway).
4.Local Windows firewall is turned off.
5.When i checked for the logs from C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log i found the error message 'The TCP output processor has paused the data flow. Forwarding to output group splunkcloud has been blocked for 598307 seconds'
Gents, can the point no.2 or point no.5 causing the issue. Anyway to fx this?
OK.We found the folder 100_CompanyName_splunkcloud is not getting create when we install the password file and it causes the issue. Thanks for your help.
Yes. If you get a prompt, then there is nothing blocking the traffic. You should be able to see your forwarder with this search:
| tstats count WHERE index=_* values(sourcetype) BY host
| search host = "Your HostName OR IP Here"
Ok. It seems i'm able to get the events by IP address
but i'm not sure i'm searching in the correct way. I queried in the below format (Source Network Address: *IP Address)*We generally search for the hostnames
by using below query index=main | stats count by host | sort -count
, May i know how do i search for the events using IP address
.
So (Source Network Address: *IP Address)* is the correct way of checking?
I do not understand what you wrote. Show me your EXACT search strings and prefix that SPL code with a blank line separating it from the rest of your text and indent each line with 4 spaces so that it gets treated as a code block by the markup renderer.
Is below query is correct way of searching for the servers by using IP address in splunk console?
Source Network Address: 10.36.128.142
I do not know what "splunk console" is. I do not know what Source Network Address: 10.36.128.142
syntax means. The only way for me to figure out is for you to POST YOUR EXACT SEARCH STRING!
This the search string Source Network Address: 10.36.128.142
If you are in Splunk Cloud, then support should have given you a 100_CompanyName_splunkcloud
app.
If you used competent PS, they should have suggest that you deploy a Deployment Server and given you an app with a deploymentclient.conf
file to point to your Deployment Server.
Both apps should go into the C:\Program Files\SplunkUniversalForwarder\etc\apps
folder. If you can identify these apps on a working forwarder, just copy them exactly as-is to the non-working forwarder and restart Splunk on the destination forwarder.
I copied the 100_CompanyName_Splunkcloud folder from the working server and restarted the service still the clients are not reporting
Can you do the telnet
test? Do you get login prompt?
I dont know what you mean exactly.
But iam able to telnet the splunk's public IP over the port 9997. Is that the thing that you are asking for?
For standard, simple, non-SSL Splunk, you need an outputs.conf
on the forwarder pointing to the Indexers and port 9997
. You should be able to telnet Your.Indexer.IP.Address 9997
and get a login prompt from the forwarder. If not, something besides Splunk is blocking.
Another observation is:
The folder 100_CompanyName_splunkcloud is not getting created when installing splunk in the path C:\Program Files\SplunkUniversalForwarder\etc\apps\
. I am able to see that folder in the reporting servers. And in that folder Pem
file, ouputs.conf
etc. exists.
This causing the issue?
@woodcock I think this is causing the issue? Do you have any thoughts on this?
Try running Splunk service as local user instead of specific account.
No account specified. Changed to local account but no luck
Hi,
maybe the official doc for trouble shooting this could add some value
https://docs.splunk.com/Documentation/Splunk/7.2.3/Troubleshooting/Cantfinddata