Deployment Architecture

Windows server not feeding logs

debjit_k
Path Finder

Hope you are doing great. 

Again facing a challenging and seeking some help.

Prob statement 

We have 200 windows server out of which 3 devices and not reporting suddenly.

I tried to check the output.conf and server.conf it looks looks fine and I also compare those files with the working server. 

Everything is fine.

And yes I check the status of the non reporting server it is showing up and running and while using TTL the server is responding not Im unable to get the data on splunk.

I don't have much idea what could be the root cause it will be great if you could suggest me something.. 

Note: Splunk installed on  on-prem 

Thanks 

Debjit 

 

Labels (3)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k ,

you can check the firewall rules using telnet from the client server

telnet ip_indexer 9997

if you haven't on that server, you have to install it.

When I spoke of Deployment server I meant: check if the three missing servers are in the server list in Deployment Server.

Then, check if the hostname in those three server is correct or if it is in also anothere connected server, you can find it in $SPLUNK_HOME/etc/apps/system/local:

  • inputs.conf
  • server.conf

Ciao.

Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k,

thank you all OK: I'm in holyday!

It's difficoult to debug your situation, anyway these are some tries:

  • did you installed all Forwarders from scratch or cloning another installation? check the hostname of these three machines to understand if it's duplicated,
  • check the connection, using telnet, between these three servers and the Indexers.
  • are these three servers using an old not certified OS version?
  • I suppose that you're using a Deployment Server to manage all your Forwarders: do you see these three servers in Deployment Server?
  • did you disabled local firewall on these servers?

Ciao.

Giuseppe

0 Karma

debjit_k
Path Finder

Hi @gcusello , 

Hope you are enjoying your holidays 

Install from scratched. 

We have deployment serve and I can see those server. 

The local Firewall is off if it is on we can't take RDP 

So not sure what is happening 

Thank you 

Debjit.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k,

and connection check with telnet?

Ciao.

Giuseppe

debjit_k
Path Finder

Hi @gcusello , 

 

How to check the telnet connection I want to verify it.

We gave the deployment server IP 8089 port is this the correct way to do so.

Thank you 

Debjit 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k ,

you can check the firewall rules using telnet from the client server

telnet ip_indexer 9997

if you haven't on that server, you have to install it.

When I spoke of Deployment server I meant: check if the three missing servers are in the server list in Deployment Server.

Then, check if the hostname in those three server is correct or if it is in also anothere connected server, you can find it in $SPLUNK_HOME/etc/apps/system/local:

  • inputs.conf
  • server.conf

Ciao.

Giuseppe

debjit_k
Path Finder

Hi @gcusello , 

Thank you for the support .

Yeah I checked the host name on the deployment server and the name is correct only..

I guess telnet is the issue.. 

So if telnet is not responding to that indexer so what is needed to install on the client-server? 

 

Thanks you 

Debjit 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k,

if you haven't telnet on those servers, you have to install it and try the check.

Ciao.

Giuseppe

0 Karma

debjit_k
Path Finder

Hi @gcusello , 

Sure.let me try 

 

Thank you 

Debjit 

0 Karma

debjit_k
Path Finder

Hi @gcusello,

Telnet is install but it is not replying to the indexer. So to fix this issue is there anything we can do from client end.

 

Thank you

Debjit 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k,

if the client, using telnet, cannot access Indexers on that port, it means that the route between them is closed, check it.

Ciao.

Giuseppe

debjit_k
Path Finder

Hi @gcusello , 

I guess from firewall end we need to check this connection correct me if im wrong.. 

 

Thanks 

Debjit 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k ,

yes, you're correct.

ciao.

Giuseppe

debjit_k
Path Finder

Hi @gcusello ,

Just one small doubt I have.

On those windows server we can see like it is sending the data to deployment server (because in output.conf it is showing deployment server IP), so why do we need to open the telnet for index server like we can only open it for deployment server It will work? 

Just a small doubt.

 

Thanks

Debjit 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k,

in "outputs.conf" you address only  the connection with the Indexers not with the Deployment Server.

The connection with the Deployment server is addressed in the "deploymentclient.conf" file.

If you're sending data to the Deployment Server you're in error, the role of DS is different: managinf Forwarders' configuration.

Ciao.

Giuseppe

0 Karma

debjit_k
Path Finder

Hi @gcusello ,

But the devices which are reporting to splunk are configure in same way.

In output.conf they gave the IP of deployment server IP and yes telnet it working for them but those 3 non reporting server the telnet is not working that's the difference.

 

Thanks 

Debjit 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @debjit_k,

if you have an all-in-one configuration, your deploymentclient.conf and outputs.conf files contain the same address because DS and IDX are the same machine.

Otherwise (when you have separated IDXs and DS) there's a configuration error: if you put the DS address in outputs.conf you use the DS as an Heavy Forwarder and it's a wrong configuration because, when DS has to manage more than 50 clients, it must has a dedicated server.

Forwarders send their logs: directly to IDXs or to one or more HFs that work as concentrators, but never to the DS!

I hint to review your architecture with a Splunk Architect.

Ciao.

Giuseppe

0 Karma

debjit_k
Path Finder

Hi @gcusello ,

We enable the telnet for indexer over the port 9997 but it is not reporting to Splunk.

According to you what will cause the issue services are running also

 

Thanks 

Debjit

0 Karma

gcusello
SplunkTrust
SplunkTrust
0 Karma

debjit_k
Path Finder

Hi @gcusello ,

Thank you for the help.

Will infome you once it is done 

Thanks 

Debjit

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...