How to monitor Splunk DB Connect V2.4 on a Search Head cluster?

[saranya_fmr]

Is there a way to monitor the status of DB Connect App on the Search head cluster?

We recently moved the DB Connect app V 1.x from a standalone SH to a clustered environment with V2.4.0
I understand that the RPC Service status in UI says if the DB Connect app is working fine and certain logs like dbx2.log , rpc.log and health.log are informative.
But is there a way to set up some kind of alerting/monitoring incase the DB Connection doesn't work i.e RPC Service is down ??

[mchang_splunk]

You can also use script input with ps -elf | grep RPCServer to monitor if RPCServer process is running.

---

. # ps -elf | grep RPCServer

0 S root 67963 67948 0 80 0 - 691125 futex_ 15:27 ? 00:00:00 /opt/jdk1.8.0_121/bin/java -XX:+UseConcMarkSweepGC -classpath /opt/splunk651/etc/apps/splunk_app_db_connect/bin/lib/rpcserver-all.jar -DSPLUNK_HOME=/opt/splunk651 com.splunk.dbx2.rpc.RPCServer 127.0.0.1:9998

[sloshburch]

I bet the ps.sh that comes with the Nix TA would nail that.

[rdagan_splunk]

This link might help: http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Troubleshooting

[saranya_fmr]

HI @rdagan ,

Th e troubleshooting doc suggests how to check any issues and fix them. I understand that I can find all the error from the Splunk log files.

But I'm trying to check if I can set up some monitoring for RPC Status. An alert and incident should be generated when the RPC service goes down so that we can monitor the DB Status in our clustered SH environment.

[sloshburch]

Hi @saranya_fmr - Sounds like you can just set up splunk searche alerts on those error messages in the respective log. Also, DBConnect includes a number of dashboards which might provide some additional color here. Have you seen them? Let us know if maybe we've misunderstood.

[saranya_fmr]

I was wondering if I could use any ticketing/alerting tool OR Splunk itself to monitor the Splunk RPC Status.
For example:

• Is there any specific process for RPC
  Status in Splunk DB Connect app , so
  that I could use port monitoring.

• I was testing in DEV Env to bring RPC
  Service down and noticed this action
  in dbx2.log
  "action=rpc_server_has_been_abnormally_terminated"
  So can I use
  rpc_server_has_been_abnormally_terminated as a keyboard to perform logfile
  monitoring on dbx2.log??

Wanted to know if this is the only keyword that occurs when RPC Service goes down?

Please provide your thoughts or any ideas on ways to monitor the RPC status.