Windows Deployments Server error on Linux Search Head

[neeravmathur]

Hi Guys,

We use 3 Search Heads (cluster-linux boxes) with 2 Deployment boxes (1-PROD, 1-QA, Win 2012R2-32GB RAM Each) as searchpeer. 

All the other servers listed under distsearch.conf of SH are linux boxes. We constantly get messages on our search head -

""Unable to distribute to peer named XXXXXXXX at uri=XXXXXXXXXX:8089 using the uri-scheme=https because peer has status=Down. Verify uri-scheme, connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.""

AND

"Problem replicating config (bundle) to search peer 'XXXXXXX', Upload bundle="/SPLUNK/splunk/var/run/54C7554E-300C-462E-A82D-6AE880CB89BF-1624948028.bundle" to peer name=XXXXXXX uri=https://XXXXXXX:8089 failed; http_status=400 http_description="Failed to untar the bundle="D:\Splunk\var\run\searchpeers\54C7554E-300C-462E-A82D-6AE880CB89BF-1624948028.bundle". This could be due Search Head attempting to upload the same bundle again after a timeout. Check for sendRcvTimeout message in splund.log, consider increasing it."."

This happens only with the 2 Win-Deployment boxes. Linux boxes do not throw such alerts ever...

My question is are both issues interrelated?
The state of these 2 servers often go from UP to DOWN on the Search peer UI on the Search Head.
Troubleshooting details below which we tried but did not work-
1. We have tried removing them and adding them again from the GUI and the distsearch.conf and authenticating them again.
2. In distsearch.conf on SH-
[replicationSettings]
sendRcvTimeout = 240
3.Size of SH bundle is about 125MB which is not huge....

Not sure what needs to be done here. Any help would be appreciated........

Hoping for a quick fix on this.
Thanks for your help.....

[gcusello]

Hi @neeravmathur,

only for information: what do you mean with  "Deployment boxes (1-PROD, 1-QA, Win 2012R2-32GB RAM Each) as searchpeer."?

are you speaking of Indexers or Deployer or Deployment Server?

If you're meaning "Deployer", in other words the Splunk component that manages the Search Head Cluster, it's better to have the same OS than the Search Heads.

Could you better describe your architecture, using the Splunk roles: Indexer, Search Head, Master Node, Deployer, Deployment Server?

Anyway, my hint is to use Windows servers at most for tests and use always Linux servers for production environments.

Ciao.

Giuseppe

[neeravmathur]

Apologies...should have been more clear.....

So IN PROD we have 3 SH (clustered), 2 Indexers (non clustered), 1 Deployer and 1 Deployment Server 

and IN QA we have 1 SH, 2 Indexers, 1 Deployment Server

Now, both the deployment Servers are Windows (having 32 GB memory) and both servers are configured in Search Head's distsearch and act as Search Peer.

All the other components like SH,Indexer,Deployer are Linux and work just fine. 

On the Search heads I always see the mentioned errors/messages.

Is there anything that I am missing or can be configured so that these sync errors do not come up...They are huge inconivence....

I agree that Linux Servers are much better but since these are deployment servers so opening ports again would be a big challenge for us.

Hope this helps...Thanks for your prompt response....

[gcusello]

Hi @neeravmathur,

I try to summarize:

in production you have:

• 3 SHs Linux clustered,
• 1 Deployer Linux that manages the clustered SHs,
• 2 Indexers Linux not clustered,
• 1 Deployment Server Windows,

In QA you have:

• 1 SH, Linux,
• 2 Indexers Linux not clustered,
• 1 Deployment Server Windows,

All the Splunk servers send their own log to the Indexers.

My first question is obviously: why do you use Windows Deployment Servers when all the other servers are Linux? I'd avoid it!

Second question: why do You use Deployment Servers as Search Peer on Search Head? it isn't an Indexer and it's a best practice that all the Splunk servers (also Deployment Servers) send log to Indexers.

Now I understand the message you have.

A correct architecture is:

• SH Cluster use both Indexers as Search Peers,
• All the servers are Linux,
• All the servers send their own logs to Indexers,
• this rules must be separately applied to Proiduction and QA Environments.

Ciao.

Giuseppe

[neeravmathur]

Hi @gcusello,

Answer#1: We have only recently setup the Linux for Splunk. So deployment servers are still Windows. Getting ports for Universal Forwarders opened is a pain...hopefully we would switch to Linux someday...

Answer#2:We have some reports that need data from the deployment server directly. Now that you have mentioned it, I might use Summary Indexing on the Deployment Servers to send the data over and disable them as search peeers. 

But until that is done, my main concern is------

Can we use a setting/config anywhere on the SH that will stop replication of bundle only on these two boxes while the bundle continues to replicate on other Linux servers?

Thanks for your help....

[gcusello · Accepted solution]

Hi @neeravmathur,

as I said, I don't like to use Deployment Server for other scope than deployment.

In addition you cannot use that Summary Index on the Search Heads.

You could send DS data to indexers and then create Summary Index on The Search Heads or the Indexers.

As I said it's a best practice that all the Splunk servers send their data to the Indexers.

Ciao.

Giuseppe

[neeravmathur]

Hi @gcusello,

Thanks for your suggestion...will try it out...thanks again for your time and help.....

[gcusello]

Hi @neeravmathur,

good for you, tell me if I can help you.

Ciao and happy splunking.

Giuseppe

P.S.: please accept the answer for the other people of Community, Karma Points are appreciated 😉