Deployment Server seen as Unavailable/Offline by other members

[romulusc]

Hi,

I'm currently stuck on an issue where my Deployment Server seems to not be communicating with the same group as the other members. The server is listed as "Offline" or "Unavailable" when accessing the Monitoring Console. I believe this may be effecting clients communicating with the deployment server receiving apps and config information. I tried to see if any services are currently not running on the Deployment Server but nothing has stood out. The Deployment Server only lists itself when I access the Monitoring Console (it's in Distributed Mode). Not sure what other details are needed as I am kind of new at this but included some pictures to help describe my situation.

Any assistance is much appreciated. Thank you.

[woodcock]

On the MC, go to Settings -> Distributed search -> Search peers and see what it tells you. Try disabling the existing entry for DS and try re-adding it. Try adding it by IP address instead of hostname (perhaps you have a DNS problem).

[woodcock]

Don’t use telnet to test SSL/TLS connections; run this from the DS client:

openssl s_client -connect deploymentserverhostname:8089

Alternatively this:

/opt/splunk/bin/splunk cmd openssl s_client -connect deploymentserverhostname:8089

[romulusc]

@woodcock

So I did that from one of the other clients and the License server and got the certificate thumbprint and the other details which should signify a successful connection using SSL/TLS

So this means there shouldn't be a communication issue right?

[woodcock]

You should either get a password prompt or a CONNECTED response.

[Vijeta]

@romulusc Check the splunkd logs or write this query and see if you find any issues. (host should be your Deployment server hostname)

index=_internal sourcetype=splunkd host=<your deployment server>

[romulusc]

Yeah I entered that query and received over 320,000+ events...I think that's a little much don't you think?
I tried using the "Interesting fields" options on the left to group "deploy-server" but that still yielded 10,000+ events just for today I think. Anything in particular I'm looking for?

[woodcock]

Check splunkd.log it is usually quite specific.

[romulusc]

Sorry but is there a specific area I am looking for in this log? Also is the one you're referring to in the \var\log directory?

[woodcock]

Here: /opt/splunk/var/log/splunk/splunkd.log
Go there and find the specific logs and report back.

[romulusc]

Again I do not know what I am looking for in this extensive log. I see a bunch of lines like this

06-11-2019 12:42:08.685 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=7 AppEvents from DC ip=10.101.201.4
name=EA8EBD62-4291-44AB-BD9C-DE50036B75BF
06-11-2019 12:42:09.376 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=7 AppEvents from DC ip=10.102.201.3
name=757FF50E-A587-4ADD-8C7A-3DF94FE6A3D9
06-11-2019 12:42:38.508 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=3 AppEvents from DC ip=10.102.204.1
name=F3273119-1099-4842-9D5D-0794B2CBE9B1
06-11-2019 12:42:38.784 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=3 AppEvents from DC ip=10.101.204.2
name=8DB575D9-937A-4BD8-B8A7-71C964BD9FBF
06-11-2019 12:44:14.292 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=7 AppEvents from DC ip=10.102.201.4
name=B1285E03-0720-4A53-BFF4-F52C17BBC4D3
06-11-2019 12:44:43.722 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=3 AppEvents from DC ip=10.101.204.3
name=CD69ADA1-AE46-4551-A27D-6975F3694934
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" Get-Item : Cannot find
path 06-11-2019 12:44:54.706 -0400
ERROR ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1"
'HKLM:\System\CurrentControlSet\Services\NTDS\Parameters'
because it does not 06-11-2019
12:44:54.706 -0400 ERROR ExecProcessor
- message from ""C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" exist. 06-11-2019
12:44:54.706 -0400 ERROR ExecProcessor
- message from ""C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" At C:\Program
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1"
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\powershell\nt6-health.ps1:66
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" char:15 06-11-2019
12:44:54.706 -0400 ERROR ExecProcessor
- message from ""C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" + $SchemaInfo =
Get-Item 06-11-2019 12:44:54.706
-0400 ERROR ExecProcessor - message from ""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1"
"HKLM:System\CurrentControlSet\Services\NTDS\Parameters"
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" +

06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" + CategoryInfo

: ObjectNotFound:
(HKLM:\System\Cu...NTDS\Paramete
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" rs:String)
[Get-Item], ItemNotFoundException
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" +
FullyQualifiedErrorId :
PathNotFound,Microsoft.PowerShell.Commands.GetIt
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" emCommand
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" 06-11-2019
12:44:54.711 -0400 ERROR ExecProcessor
- message from ""C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" You cannot call a
method on a null-valued expression.

Does this mean anything to you?

[satyaallaparthi]

Hello,

Can you go to monitoring console-->Settings and general setup.. Distributed and reset---> apply and restart server.. might be that will work.

Thanks,

[romulusc]

Am I doing the "Reset All Settings" on the Deployment Server? That'll remove all of the Forward Management and all that stuff correct?

[satyaallaparthi]

can you share the screen shot if u can?
yes in deployment server..monitoring console..general setup and distributed and then reset there and apply settings..

[romulusc]

Well I did that and it is still showing the same status. And yes I stopped and started the Splunk service

[satyaallaparthi]

did you check all firewalls or ports or communication between those server.. do telnet and see if you can find something..