All Apps and Add-ons

How to monitor Splunk DB Connect V2.4 on a Search Head cluster?

saranya_fmr
Communicator

Is there a way to monitor the status of DB Connect App on the Search head cluster?

We recently moved the DB Connect app V 1.x from a standalone SH to a clustered environment with V2.4.0
I understand that the RPC Service status in UI says if the DB Connect app is working fine and certain logs like dbx2.log , rpc.log and health.log are informative.
But is there a way to set up some kind of alerting/monitoring incase the DB Connection doesn't work i.e RPC Service is down ??

0 Karma

mchang_splunk
Splunk Employee
Splunk Employee

You can also use script input with ps -elf | grep RPCServer to monitor if RPCServer process is running.


. # ps -elf | grep RPCServer

0 S root 67963 67948 0 80 0 - 691125 futex_ 15:27 ? 00:00:00 /opt/jdk1.8.0_121/bin/java -XX:+UseConcMarkSweepGC -classpath /opt/splunk651/etc/apps/splunk_app_db_connect/bin/lib/rpcserver-all.jar -DSPLUNK_HOME=/opt/splunk651 com.splunk.dbx2.rpc.RPCServer 127.0.0.1:9998

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I bet the ps.sh that comes with the Nix TA would nail that.

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee
0 Karma

saranya_fmr
Communicator

HI @rdagan ,

Th e troubleshooting doc suggests how to check any issues and fix them. I understand that I can find all the error from the Splunk log files.

But I'm trying to check if I can set up some monitoring for RPC Status. An alert and incident should be generated when the RPC service goes down so that we can monitor the DB Status in our clustered SH environment.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Hi @saranya_fmr - Sounds like you can just set up splunk searche alerts on those error messages in the respective log. Also, DBConnect includes a number of dashboards which might provide some additional color here. Have you seen them? Let us know if maybe we've misunderstood.

0 Karma

saranya_fmr
Communicator

I was wondering if I could use any ticketing/alerting tool OR Splunk itself to monitor the Splunk RPC Status.
For example:

  • Is there any specific process for RPC
    Status in Splunk DB Connect app , so
    that I could use port monitoring.

  • I was testing in DEV Env to bring RPC
    Service down and noticed this action
    in dbx2.log
    "action=rpc_server_has_been_abnormally_terminated"
    So can I use
    rpc_server_has_been_abnormally_terminated as a keyboard to perform logfile
    monitoring on dbx2.log??

Wanted to know if this is the only keyword that occurs when RPC Service goes down?

Please provide your thoughts or any ideas on ways to monitor the RPC status.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...