Windows server not feeding logs

[debjit_k]

Hope you are doing great. 

Again facing a challenging and seeking some help.

Prob statement 

We have 200 windows server out of which 3 devices and not reporting suddenly.

I tried to check the output.conf and server.conf it looks looks fine and I also compare those files with the working server. 

Everything is fine.

And yes I check the status of the non reporting server it is showing up and running and while using TTL the server is responding not Im unable to get the data on splunk.

I don't have much idea what could be the root cause it will be great if you could suggest me something.. 

Note: Splunk installed on  on-prem 

Thanks 

Debjit 

[gcusello]

Hi @debjit_k,

thank you all OK: I'm in holyday!

It's difficoult to debug your situation, anyway these are some tries:

• did you installed all Forwarders from scratch or cloning another installation? check the hostname of these three machines to understand if it's duplicated,
• check the connection, using telnet, between these three servers and the Indexers.
• are these three servers using an old not certified OS version?
• I suppose that you're using a Deployment Server to manage all your Forwarders: do you see these three servers in Deployment Server?
• did you disabled local firewall on these servers?

Ciao.

Giuseppe

[debjit_k]

Hi @gcusello , 

Hope you are enjoying your holidays 

Install from scratched. 

We have deployment serve and I can see those server. 

The local Firewall is off if it is on we can't take RDP 

So not sure what is happening 

Thank you 

Debjit.

[gcusello]

Hi @debjit_k,

and connection check with telnet?

Ciao.

Giuseppe

[debjit_k]

Hi @gcusello , 

How to check the telnet connection I want to verify it.

We gave the deployment server IP 8089 port is this the correct way to do so.

Thank you 

Debjit 

[gcusello · Accepted solution]

Hi @debjit_k ,

you can check the firewall rules using telnet from the client server

telnet ip_indexer 9997

if you haven't on that server, you have to install it.

When I spoke of Deployment server I meant: check if the three missing servers are in the server list in Deployment Server.

Then, check if the hostname in those three server is correct or if it is in also anothere connected server, you can find it in $SPLUNK_HOME/etc/apps/system/local:

• inputs.conf
• server.conf

Ciao.

Giuseppe

[debjit_k]

Hi @gcusello , 

Thank you for the support .

Yeah I checked the host name on the deployment server and the name is correct only..

I guess telnet is the issue.. 

So if telnet is not responding to that indexer so what is needed to install on the client-server? 

Thanks you 

Debjit 

[gcusello]

Hi @debjit_k,

if you haven't telnet on those servers, you have to install it and try the check.

Ciao.

Giuseppe

[debjit_k]

Hi @gcusello , 

Sure.let me try 

Thank you 

Debjit 

[debjit_k]

Hi @gcusello,

Telnet is install but it is not replying to the indexer. So to fix this issue is there anything we can do from client end.

Thank you

Debjit 

[gcusello]

Hi @debjit_k,

if the client, using telnet, cannot access Indexers on that port, it means that the route between them is closed, check it.

Ciao.

Giuseppe

[debjit_k]

Hi @gcusello , 

I guess from firewall end we need to check this connection correct me if im wrong.. 

Thanks 

Debjit 

[gcusello]

Hi @debjit_k ,

yes, you're correct.

ciao.

Giuseppe

[debjit_k]

Hi @gcusello ,

Just one small doubt I have.

On those windows server we can see like it is sending the data to deployment server (because in output.conf it is showing deployment server IP), so why do we need to open the telnet for index server like we can only open it for deployment server It will work? 

Just a small doubt.

Thanks

Debjit 

[gcusello]

Hi @debjit_k,

in "outputs.conf" you address only  the connection with the Indexers not with the Deployment Server.

The connection with the Deployment server is addressed in the "deploymentclient.conf" file.

If you're sending data to the Deployment Server you're in error, the role of DS is different: managinf Forwarders' configuration.

Ciao.

Giuseppe

[debjit_k]

Hi @gcusello ,

But the devices which are reporting to splunk are configure in same way.

In output.conf they gave the IP of deployment server IP and yes telnet it working for them but those 3 non reporting server the telnet is not working that's the difference.

Thanks 

Debjit 

[gcusello]

Hi @debjit_k,

if you have an all-in-one configuration, your deploymentclient.conf and outputs.conf files contain the same address because DS and IDX are the same machine.

Otherwise (when you have separated IDXs and DS) there's a configuration error: if you put the DS address in outputs.conf you use the DS as an Heavy Forwarder and it's a wrong configuration because, when DS has to manage more than 50 clients, it must has a dedicated server.

Forwarders send their logs: directly to IDXs or to one or more HFs that work as concentrators, but never to the DS!

I hint to review your architecture with a Splunk Architect.

Ciao.

Giuseppe

[debjit_k]

Hi @gcusello ,

We enable the telnet for indexer over the port 9997 but it is not reporting to Splunk.

According to you what will cause the issue services are running also

Thanks 

Debjit

[gcusello]

Hi @debjit_k,

for more infos see at https://www.splunk.com/pdfs/technical-briefs/splunk-validated-architectures.pdf

Ciao.

Giuseppe

[debjit_k]

Hi @gcusello ,

Thank you for the help.

Will infome you once it is done 

Thanks 

Debjit