All Apps and Add-ons

Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

bworrellZP
Communicator

Splunk DB Connect is not populating data on the Health screen. Anyone know the cause of this or how to fix / look into it?

1 Solution

bworrellZP
Communicator

Answer is - Seems there is a Bug in the DB Connect client since 2.0.5. When the py_health.log hits 10,000 kb, the file locks, it does not rollover or purge as it should. That causes errors in the splunkd.log about mi_input.py being in use by another process.

Once I purged part of the file, the health data showed up.

Had nothing to do with the suggestion by jcoates (hence why I was able to get Health stats prior to 2.0.5.)

Just a bug in the DB Connect client. Will watch the log and clean it out until issue is fixed.

View solution in original post

bworrellZP
Communicator

Answer is - Seems there is a Bug in the DB Connect client since 2.0.5. When the py_health.log hits 10,000 kb, the file locks, it does not rollover or purge as it should. That causes errors in the splunkd.log about mi_input.py being in use by another process.

Once I purged part of the file, the health data showed up.

Had nothing to do with the suggestion by jcoates (hence why I was able to get Health stats prior to 2.0.5.)

Just a bug in the DB Connect client. Will watch the log and clean it out until issue is fixed.

HMTODD
Explorer

Thanks you for this! Just ran into the same problem. Purged records from the py_health.log file and dashboard began to populate. Anyone know if Splunk intended to fix this bug?

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

meta... converted a couple of the clarification request comments into answers to clear the un-answered flag.

0 Karma

bworrellZP
Communicator

So, issue is not resolved, comments did not offer any suggestions.

Looking elsewhere, found the log path. Seems that the py_health.log file hasn't had any updates in almost two months. Since I updated to 2.0.4.

Seems the health dashboard needs that log to update.

Anyone else have this issue?

0 Karma

stmyers7941
Path Finder

Click the "RPC Service: Up" icon from the menu, which will launch a search for errors in RPC, dbx2 and dbx_health. Make sure you have values for dbx_health.

0 Karma

bworrellZP
Communicator

Have data for all three source types.

In the Health tab, I selected the Connections and opened the query in a search window. Comes up with the below.

| tstats values(All_Records.CONNECTION) as CONNECTION values(All_Records.DB_USER) as DB_USER values(All_Records.FUNCTION) as FUNCTION values(All_Records.STATE) as STATE from datamodel=DB_Health groupby "All_Records.TRANS_OBJECT_ID" | rename All_Records.* as * | eval FUNCTION=mvfilter(match(FUNCTION, "py_*"))  | `health_transaction_filter(*, *, *)`| stats count as Total count(eval(mvfind(STATE,"error"))) as Failed by CONNECTION | eval "% Failed" = (Failed*100/Total) | eval Successful=(Total-Failed) | table CONNECTION Total Failed Successful "% Failed"

That tells me there are 3000 events, but no results found.

Pairing it down to this:

| tstats values(All_Records.CONNECTION) as CONNECTION values(All_Records.DB_USER) as DB_USER values(All_Records.FUNCTION) as FUNCTION values(All_Records.STATE) as STATE from datamodel=DB_Health groupby "All_Records.TRANS_OBJECT_ID" | rename All_Records.* as * | eval FUNCTION=mvfilter(match(FUNCTION, "py_*"))

gets me at lease some stats, though not much. Still 3000 events, but no results.

0 Karma

stmyers7941
Path Finder

Is this a distributed environment? Is DBConnect running on your search head?

0 Karma

bworrellZP
Communicator

No DB Connect is not running on my search head. I was on the Windows Indexer I have, and it was on this indexer that I went into the health tab in the DB Connect App. This was working in 2.03.

I can install DB Connect on my search head, but do not recall that being a requirement from the install doc.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

you need the data model to use the health page.

The docs could be clearer... http://docs.splunk.com/Documentation/DBX/2.0.6/DeployDBX/Architectureandperformanceconsiderations

There's four ways to use DB Connect.
A) ingest data from an RDBMS into a Splunk index. A heavy forwarder is ideal, but technically it could work on search heads or indexers.
B) ingest data from an RDBMS into a Splunk lookup. A search head is ideal, because your users doing searching are the ones using it.
C) export data from Splunk into an RDBMS (e.g. making a table of churn predictions or security risk values). A heavy forwarder is ideal, but technically it could work on search heads or indexers.
D) Use Splunk as an interactive SQL reporting tool. A search head is ideal, but technically it could work on indexers or heavy forwarders.

0 Karma

bworrellZP
Communicator

Jcoates,

When I did the original install, there was a note that IF you where pushing out the app via a deployment server, that it had to be on the search head. That said, even in this document, once it is installed all the work is done by the indexer, as I have it.

Also, prior to the update to 2.04, the health tab worked fine.

So I have installed the app on the search head now. Still NOT using the deployment app portion, as there does not appear to be a need based on the document. I only have one server doing database pulls. Data pulls in fine (one issue but unrelated to the app).

Now with that said, any thoughts as to why neither show the Health tab or where to look to find the errors?

Thanks

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

Do you have a valid connection to your DB? what do you see in $SPLUNK_HOME/var/log/splunk/dbx2.log ?

0 Karma

bworrellZP
Communicator

Have a connection, pulling in data from multiple servers just fine. Log file is populating, just not as much as I would expect for the data I am pulling in.

Before I recall an error in the dashboard about health.py, but that does not happen anymore (did update from 2.04 to 2.05), now the dashboard drop downs have no data.

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

Do you get any results when you run this search?

| tstats dc(All_Records.CONNECTION) from datamodel=DB_Health groupby "All_Records.CONNECTION" | `rename_all`

bworrellZP
Communicator

I get Error in 'TsidxStats': Could not find datamodel: DB_Health

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...