topic Re: Windows server not feeding logs in Deployment Architecture

Windows server not feeding logs

debjit_k — Tue, 16 Aug 2022 14:13:32 GMT

Hope you are doing great.

Again facing a challenging and seeking some help.

Prob statement

We have 200 windows server out of which 3 devices and not reporting suddenly.

I tried to check the output.conf and server.conf it looks looks fine and I also compare those files with the working server.

Everything is fine.

And yes I check the status of the non reporting server it is showing up and running and while using TTL the server is responding not Im unable to get the data on splunk.

I don't have much idea what could be the root cause it will be great if you could suggest me something..

Note: Splunk installed on on-prem

Thanks

Debjit

Re: Windows server not feeding logs

gcusello — Sun, 31 Jul 2022 14:04:21 GMT

Hi @debjit_k,

thank you all OK: I'm in holyday!

It's difficoult to debug your situation, anyway these are some tries:

did you installed all Forwarders from scratch or cloning another installation? check the hostname of these three machines to understand if it's duplicated,
check the connection, using telnet, between these three servers and the Indexers.
are these three servers using an old not certified OS version?
I suppose that you're using a Deployment Server to manage all your Forwarders: do you see these three servers in Deployment Server?
did you disabled local firewall on these servers?

Ciao.

Giuseppe

Re: Windows server not feeding logs

debjit_k — Sun, 31 Jul 2022 14:33:51 GMT

Hi @gcusello ,

Hope you are enjoying your holidays

Install from scratched.

We have deployment serve and I can see those server.

The local Firewall is off if it is on we can't take RDP

So not sure what is happening

Thank you

Debjit.

Re: Windows server not feeding logs

gcusello — Sun, 31 Jul 2022 14:35:42 GMT

Hi @debjit_k,

and connection check with telnet?

Ciao.

Giuseppe

Re: Windows server not feeding logs

debjit_k — Sun, 31 Jul 2022 14:51:03 GMT

Hi @gcusello ,

How to check the telnet connection I want to verify it.

We gave the deployment server IP 8089 port is this the correct way to do so.

Thank you

Debjit

Re: Windows server not feeding logs

gcusello — Sun, 31 Jul 2022 14:56:09 GMT

Hi @debjit_k ,

you can check the firewall rules using telnet from the client server

telnet ip_indexer 9997

if you haven't on that server, you have to install it.

When I spoke of Deployment server I meant: check if the three missing servers are in the server list in Deployment Server.

Then, check if the hostname in those three server is correct or if it is in also anothere connected server, you can find it in $SPLUNK_HOME/etc/apps/system/local:

inputs.conf
server.conf

Ciao.

Giuseppe

Re: Windows server not feeding logs

debjit_k — Sun, 31 Jul 2022 15:04:44 GMT

Hi @gcusello ,

Thank you for the support .

Yeah I checked the host name on the deployment server and the name is correct only..

I guess telnet is the issue..

So if telnet is not responding to that indexer so what is needed to install on the client-server?

Thanks you

Debjit

Re: Windows server not feeding logs

gcusello — Sun, 31 Jul 2022 15:06:40 GMT

Hi @debjit_k,

if you haven't telnet on those servers, you have to install it and try the check.

Ciao.

Giuseppe

Re: Windows server not feeding logs

debjit_k — Sun, 31 Jul 2022 15:08:36 GMT

Hi @gcusello ,

Sure.let me try

Thank you

Debjit

Re: Windows server not feeding logs

debjit_k — Sun, 31 Jul 2022 15:15:14 GMT

Hi @gcusello,

Telnet is install but it is not replying to the indexer. So to fix this issue is there anything we can do from client end.

Thank you

Debjit

Re: Windows server not feeding logs

gcusello — Sun, 31 Jul 2022 15:17:18 GMT

Hi @debjit_k,

if the client, using telnet, cannot access Indexers on that port, it means that the route between them is closed, check it.

Ciao.

Giuseppe

Re: Windows server not feeding logs

debjit_k — Sun, 31 Jul 2022 15:19:59 GMT

Hi @gcusello ,

I guess from firewall end we need to check this connection correct me if im wrong..

Thanks

Debjit

Re: Windows server not feeding logs

gcusello — Sun, 31 Jul 2022 15:32:15 GMT

Hi @debjit_k ,

yes, you're correct.

ciao.

Giuseppe

Re: Windows server not feeding logs

debjit_k — Sun, 31 Jul 2022 15:58:37 GMT

Hi @gcusello ,

Thank you for the help.

Will infome you once it is done

Thanks

Debjit

Re: Windows server not feeding logs

debjit_k — Mon, 01 Aug 2022 03:55:51 GMT

Hi @gcusello ,

Just one small doubt I have.

On those windows server we can see like it is sending the data to deployment server (because in output.conf it is showing deployment server IP), so why do we need to open the telnet for index server like we can only open it for deployment server It will work?

Just a small doubt.

Thanks

Debjit

Re: Windows server not feeding logs

gcusello — Mon, 01 Aug 2022 06:38:41 GMT

Hi @debjit_k,

in "outputs.conf" you address only the connection with the Indexers not with the Deployment Server.

The connection with the Deployment server is addressed in the "deploymentclient.conf" file.

If you're sending data to the Deployment Server you're in error, the role of DS is different: managinf Forwarders' configuration.

Ciao.

Giuseppe

Re: Windows server not feeding logs

debjit_k — Mon, 01 Aug 2022 07:49:19 GMT

Hi @gcusello ,

But the devices which are reporting to splunk are configure in same way.

In output.conf they gave the IP of deployment server IP and yes telnet it working for them but those 3 non reporting server the telnet is not working that's the difference.

Thanks

Debjit

Re: Windows server not feeding logs

gcusello — Mon, 01 Aug 2022 08:02:33 GMT

Hi @debjit_k,

if you have an all-in-one configuration, your deploymentclient.conf and outputs.conf files contain the same address because DS and IDX are the same machine.

Otherwise (when you have separated IDXs and DS) there's a configuration error: if you put the DS address in outputs.conf you use the DS as an Heavy Forwarder and it's a wrong configuration because, when DS has to manage more than 50 clients, it must has a dedicated server.

Forwarders send their logs: directly to IDXs or to one or more HFs that work as concentrators, but never to the DS!

I hint to review your architecture with a Splunk Architect.

Ciao.

Giuseppe

Re: Windows server not feeding logs

gcusello — Mon, 01 Aug 2022 08:04:11 GMT

Hi @debjit_k,

for more infos see at https://www.splunk.com/pdfs/technical-briefs/splunk-validated-architectures.pdf

Ciao.

Giuseppe

Re: Windows server not feeding logs

debjit_k — Mon, 01 Aug 2022 12:47:34 GMT

Hi @gcusello ,

We enable the telnet for indexer over the port 9997 but it is not reporting to Splunk.

According to you what will cause the issue services are running also

Thanks

Debjit