topic Re: Splunk UBA is down in Splunk User Behavior Analytics

Splunk UBA is down

snisaxena — Fri, 03 Sep 2021 06:02:44 GMT

Splunk UBA search head is down.

Even after restarting ui services, status is shown as active in CLI but GUI is not available.

Commands used to stop/start ui service:

sudo service caspida-ui stop
sudo service caspida-ui start

Status when checked in CLI:

● caspida-ui.service
Loaded: loaded (/etc/init.d/caspida-ui; bad; vendor preset: enabled)
Active: active (exited) since Fri 2021-09-03 05:53:12 UTC; 6min ago

I also tried rebooting the VM, but it doesn't help.

Can I please get a suggestion around how to fix this?

Re: Splunk UBA is down

lakshman239 — Fri, 03 Sep 2021 12:11:31 GMT

@snisaxena One option would be stop and start all services, so they start gracefully. Pls refer to - https://docs.splunk.com/Documentation/UBA/5.0.4.1/Admin/CLICommands

Re: Splunk UBA is down

snisaxena — Fri, 03 Sep 2021 13:13:53 GMT

@lakshman239I ran /opt/caspida/bin/Caspida stop-all and it has been running since more than 2 hours now.
I tried to exit and run /opt/caspida/bin/Caspida start-all. It was aborted with below message:

failed to check/update system configuration: aborting. see /var/vcap/sys/log/caspida/caspida.out

Re: Splunk UBA is down

lakshman239 — Fri, 03 Sep 2021 13:24:46 GMT

stop-all running for long time does indicate an underlying issue in the cluster.

Have you run the pre-check and post health checks using the latest available scripts? If not, please run them and perhaps raise a case with support attaching the output.

Re: Splunk UBA is down

snisaxena — Fri, 03 Sep 2021 13:28:03 GMT

@lakshman239 I did run a health check before running stop-all and observed below error:

ui connect: <hostname> <= curl failed to ui <hostname>
curl: (7) Failed to connect to <hostname> port 443: Connection refused
ui connect: sc2-splunk-uba-1 <= curl failed to ui <hostname>
curl: (7) Failed to connect to <hostname> port 443: Connection refused

Re: Splunk UBA is down

lakshman239 — Fri, 03 Sep 2021 13:31:20 GMT

did this setup work in the past? If so, has there been any changes to IP/host/dns resolution and/or firewall/connectivity? looks like connectivity/resolution issue

Re: Splunk UBA is down

snisaxena — Fri, 03 Sep 2021 13:42:18 GMT

@lakshman239 I suspect so too. However, there is no confirmation from network team regarding any connection changes wrt firewall, etc.