Monitoring Splunk

Client not reporting in Splunk

sugandhakumar
New Member

Two of my servers not reporting in Splunk. They are running in windows server 2012 r2 std and 2016 datacenter. Splunk universal forwarder 7.2.0 installed in both servers.

Please find find my below observations:

1.Iam able to telnet the below IPs.
telnet 54.157.x.x 9997
telnet 34.197.x.x 9997
telnet 35.175.x.x 9997
telnet 54.241.x.x 443
2.So port is allowed but when i run netstat -a 9997 port not shows.
3.Splunk service is running in both servers(But when i try to restart, first time it shows error (windows cannot stop splunk forwarder service on local computer Error:1053) and the service gets stopped but iam able to start the service anyway).
4.Local Windows firewall is turned off.
5.When i checked for the logs from C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log i found the error message 'The TCP output processor has paused the data flow. Forwarding to output group splunkcloud has been blocked for 598307 seconds'

Gents, can the point no.2 or point no.5 causing the issue. Anyway to fx this?

Tags (1)
0 Karma

sugandhakumar
New Member

OK.We found the folder 100_CompanyName_splunkcloud is not getting create when we install the password file and it causes the issue. Thanks for your help.

0 Karma

woodcock
Esteemed Legend

Yes. If you get a prompt, then there is nothing blocking the traffic. You should be able to see your forwarder with this search:

| tstats count WHERE index=_* values(sourcetype) BY host
| search host = "Your HostName OR IP Here"
0 Karma

sugandhakumar
New Member

Ok. It seems i'm able to get the events by IP address but i'm not sure i'm searching in the correct way. I queried in the below format (Source Network Address: *IP Address)*We generally search for the hostnames by using below query index=main | stats count by host | sort -count, May i know how do i search for the events using IP address.

So (Source Network Address: *IP Address)* is the correct way of checking?

0 Karma

woodcock
Esteemed Legend

I do not understand what you wrote. Show me your EXACT search strings and prefix that SPL code with a blank line separating it from the rest of your text and indent each line with 4 spaces so that it gets treated as a code block by the markup renderer.

0 Karma

sugandhakumar
New Member

Is below query is correct way of searching for the servers by using IP address in splunk console?
Source Network Address: 10.36.128.142

0 Karma

woodcock
Esteemed Legend

I do not know what "splunk console" is. I do not know what Source Network Address: 10.36.128.142 syntax means. The only way for me to figure out is for you to POST YOUR EXACT SEARCH STRING!

0 Karma

sugandhakumar
New Member

This the search string Source Network Address: 10.36.128.142

0 Karma

woodcock
Esteemed Legend

If you are in Splunk Cloud, then support should have given you a 100_CompanyName_splunkcloud app.
If you used competent PS, they should have suggest that you deploy a Deployment Server and given you an app with a deploymentclient.conf file to point to your Deployment Server.

Both apps should go into the C:\Program Files\SplunkUniversalForwarder\etc\apps folder. If you can identify these apps on a working forwarder, just copy them exactly as-is to the non-working forwarder and restart Splunk on the destination forwarder.

0 Karma

sugandhakumar
New Member

I copied the 100_CompanyName_Splunkcloud folder from the working server and restarted the service still the clients are not reporting

0 Karma

woodcock
Esteemed Legend

Can you do the telnet test? Do you get login prompt?

0 Karma

sugandhakumar
New Member

I dont know what you mean exactly.

But iam able to telnet the splunk's public IP over the port 9997. Is that the thing that you are asking for?

0 Karma

woodcock
Esteemed Legend

For standard, simple, non-SSL Splunk, you need an outputs.conf on the forwarder pointing to the Indexers and port 9997. You should be able to telnet Your.Indexer.IP.Address 9997 and get a login prompt from the forwarder. If not, something besides Splunk is blocking.

0 Karma

sugandhakumar
New Member

Another observation is:

The folder 100_CompanyName_splunkcloud is not getting created when installing splunk in the path C:\Program Files\SplunkUniversalForwarder\etc\apps\. I am able to see that folder in the reporting servers. And in that folder Pem file, ouputs.conf etc. exists.

This causing the issue?

0 Karma

sugandhakumar
New Member

@woodcock I think this is causing the issue? Do you have any thoughts on this?

0 Karma

saurabh009
Path Finder

Try running Splunk service as local user instead of specific account.

0 Karma

sugandhakumar
New Member

No account specified. Changed to local account but no luck

0 Karma

dkeck
Influencer

Hi,

maybe the official doc for trouble shooting this could add some value

https://docs.splunk.com/Documentation/Splunk/7.2.3/Troubleshooting/Cantfinddata

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...