topic Re: Client not reporting in Splunk in Monitoring Splunk

Client not reporting in Splunk

sugandhakumar — Tue, 29 Jan 2019 17:52:31 GMT

Two of my servers not reporting in Splunk. They are running in windows server 2012 r2 std and 2016 datacenter. Splunk universal forwarder 7.2.0 installed in both servers.

Please find find my below observations:

1.Iam able to telnet the below IPs.
telnet 54.157.x.x 9997
telnet 34.197.x.x 9997
telnet 35.175.x.x 9997
telnet 54.241.x.x 443
2.So port is allowed but when i run netstat -a 9997 port not shows.
3.Splunk service is running in both servers(But when i try to restart, first time it shows error (windows cannot stop splunk forwarder service on local computer Error:1053) and the service gets stopped but iam able to start the service anyway).
4.Local Windows firewall is turned off.
5.When i checked for the logs from C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log i found the error message 'The TCP output processor has paused the data flow. Forwarding to output group splunkcloud has been blocked for 598307 seconds'

Gents, can the point no.2 or point no.5 causing the issue. Anyway to fx this?

Re: Client not reporting in Splunk

dkeck — Tue, 29 Jan 2019 19:04:23 GMT

Hi,

maybe the official doc for trouble shooting this could add some value

https://docs.splunk.com/Documentation/Splunk/7.2.3/Troubleshooting/Cantfinddata

Re: Client not reporting in Splunk

saurabh009 — Tue, 29 Jan 2019 19:25:59 GMT

Try running Splunk service as local user instead of specific account.

Re: Client not reporting in Splunk

sugandhakumar — Tue, 29 Jan 2019 20:05:15 GMT

No account specified. Changed to local account but no luck

Re: Client not reporting in Splunk

sugandhakumar — Tue, 29 Sep 2020 22:58:57 GMT

Another observation is:

The folder 100_CompanyName_splunkcloud is not getting created when installing splunk in the path C:\Program Files\SplunkUniversalForwarder\etc\apps\. I am able to see that folder in the reporting servers. And in that folder Pem file, ouputs.conf etc. exists.

This causing the issue?

Re: Client not reporting in Splunk

woodcock — Wed, 30 Jan 2019 01:00:43 GMT

For standard, simple, non-SSL Splunk, you need an outputs.conf on the forwarder pointing to the Indexers and port 9997. You should be able to telnet Your.Indexer.IP.Address 9997 and get a login prompt from the forwarder. If not, something besides Splunk is blocking.

Re: Client not reporting in Splunk

sugandhakumar — Wed, 30 Jan 2019 14:01:39 GMT

@woodcock I think this is causing the issue? Do you have any thoughts on this?

Re: Client not reporting in Splunk

woodcock — Wed, 30 Jan 2019 15:01:30 GMT

If you are in Splunk Cloud, then support should have given you a 100_CompanyName_splunkcloud app.
If you used competent PS, they should have suggest that you deploy a Deployment Server and given you an app with a deploymentclient.conf file to point to your Deployment Server.

Both apps should go into the C:\Program Files\SplunkUniversalForwarder\etc\apps folder. If you can identify these apps on a working forwarder, just copy them exactly as-is to the non-working forwarder and restart Splunk on the destination forwarder.

Re: Client not reporting in Splunk

sugandhakumar — Tue, 29 Sep 2020 23:01:15 GMT

I copied the 100_CompanyName_Splunkcloud folder from the working server and restarted the service still the clients are not reporting

Re: Client not reporting in Splunk

woodcock — Wed, 30 Jan 2019 16:49:27 GMT

Can you do the telnet test? Do you get login prompt?

Re: Client not reporting in Splunk

sugandhakumar — Wed, 30 Jan 2019 17:29:29 GMT

I dont know what you mean exactly.

But iam able to telnet the splunk's public IP over the port 9997. Is that the thing that you are asking for?

Re: Client not reporting in Splunk

woodcock — Wed, 30 Jan 2019 17:35:12 GMT

Yes. If you get a prompt, then there is nothing blocking the traffic. You should be able to see your forwarder with this search:

| tstats count WHERE index=_* values(sourcetype) BY host
| search host = "Your HostName OR IP Here"

Re: Client not reporting in Splunk

sugandhakumar — Wed, 30 Jan 2019 18:15:38 GMT

Ok. It seems i'm able to get the events by IP address but i'm not sure i'm searching in the correct way. I queried in the below format (Source Network Address: *IP Address)*We generally search for the hostnames by using below query index=main | stats count by host | sort -count, May i know how do i search for the events using IP address.

So (Source Network Address: *IP Address)* is the correct way of checking?

Re: Client not reporting in Splunk

woodcock — Wed, 30 Jan 2019 18:51:42 GMT

I do not understand what you wrote. Show me your EXACT search strings and prefix that SPL code with a blank line separating it from the rest of your text and indent each line with 4 spaces so that it gets treated as a code block by the markup renderer.

Re: Client not reporting in Splunk

sugandhakumar — Wed, 30 Jan 2019 20:22:17 GMT

Is below query is correct way of searching for the servers by using IP address in splunk console?
Source Network Address: 10.36.128.142

Re: Client not reporting in Splunk

woodcock — Wed, 30 Jan 2019 20:50:16 GMT

I do not know what "splunk console" is. I do not know what Source Network Address: 10.36.128.142 syntax means. The only way for me to figure out is for you to POST YOUR EXACT SEARCH STRING!

Re: Client not reporting in Splunk

sugandhakumar — Wed, 30 Jan 2019 20:53:41 GMT

This the search string Source Network Address: 10.36.128.142

Re: Client not reporting in Splunk

sugandhakumar — Tue, 29 Sep 2020 23:03:00 GMT

OK.We found the folder 100_CompanyName_splunkcloud is not getting create when we install the password file and it causes the issue. Thanks for your help.