topic Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data? in All Apps and Add-ons

Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

bworrellZP — Wed, 14 Oct 2015 14:47:53 GMT

Splunk DB Connect is not populating data on the Health screen. Anyone know the cause of this or how to fix / look into it?

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

rphillips_splk — Wed, 14 Oct 2015 14:57:39 GMT

Do you have a valid connection to your DB? what do you see in $SPLUNK_HOME/var/log/splunk/dbx2.log ?

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

bworrellZP — Wed, 14 Oct 2015 15:39:03 GMT

Have a connection, pulling in data from multiple servers just fine. Log file is populating, just not as much as I would expect for the data I am pulling in.

Before I recall an error in the dashboard about health.py, but that does not happen anymore (did update from 2.04 to 2.05), now the dashboard drop downs have no data.

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

rphillips_splk — Wed, 14 Oct 2015 15:52:24 GMT

Do you get any results when you run this search?

| tstats dc(All_Records.CONNECTION) from datamodel=DB_Health groupby "All_Records.CONNECTION" | `rename_all`

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

bworrellZP — Wed, 14 Oct 2015 16:03:43 GMT

I get Error in 'TsidxStats': Could not find datamodel: DB_Health

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

stmyers7941 — Tue, 29 Sep 2020 07:37:57 GMT

Click the "RPC Service: Up" icon from the menu, which will launch a search for errors in RPC, dbx2 and dbx_health. Make sure you have values for dbx_health.

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

bworrellZP — Wed, 14 Oct 2015 16:16:53 GMT

Have data for all three source types.

In the Health tab, I selected the Connections and opened the query in a search window. Comes up with the below.

| tstats values(All_Records.CONNECTION) as CONNECTION values(All_Records.DB_USER) as DB_USER values(All_Records.FUNCTION) as FUNCTION values(All_Records.STATE) as STATE from datamodel=DB_Health groupby "All_Records.TRANS_OBJECT_ID" | rename All_Records.* as * | eval FUNCTION=mvfilter(match(FUNCTION, "py_*"))  | `health_transaction_filter(*, *, *)`| stats count as Total count(eval(mvfind(STATE,"error"))) as Failed by CONNECTION | eval "% Failed" = (Failed*100/Total) | eval Successful=(Total-Failed) | table CONNECTION Total Failed Successful "% Failed"

That tells me there are 3000 events, but no results found.

Pairing it down to this:

| tstats values(All_Records.CONNECTION) as CONNECTION values(All_Records.DB_USER) as DB_USER values(All_Records.FUNCTION) as FUNCTION values(All_Records.STATE) as STATE from datamodel=DB_Health groupby "All_Records.TRANS_OBJECT_ID" | rename All_Records.* as * | eval FUNCTION=mvfilter(match(FUNCTION, "py_*"))

gets me at lease some stats, though not much. Still 3000 events, but no results.

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

stmyers7941 — Wed, 14 Oct 2015 17:22:55 GMT

Is this a distributed environment? Is DBConnect running on your search head?

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

bworrellZP — Wed, 14 Oct 2015 17:30:49 GMT

No DB Connect is not running on my search head. I was on the Windows Indexer I have, and it was on this indexer that I went into the health tab in the DB Connect App. This was working in 2.03.

I can install DB Connect on my search head, but do not recall that being a requirement from the install doc.

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

jcoates_splunk — Sat, 17 Oct 2015 20:52:18 GMT

you need the data model to use the health page.

The docs could be clearer... http://docs.splunk.com/Documentation/DBX/2.0.6/DeployDBX/Architectureandperformanceconsiderations

There's four ways to use DB Connect.
A) ingest data from an RDBMS into a Splunk index. A heavy forwarder is ideal, but technically it could work on search heads or indexers.
B) ingest data from an RDBMS into a Splunk lookup. A search head is ideal, because your users doing searching are the ones using it.
C) export data from Splunk into an RDBMS (e.g. making a table of churn predictions or security risk values). A heavy forwarder is ideal, but technically it could work on search heads or indexers.
D) Use Splunk as an interactive SQL reporting tool. A search head is ideal, but technically it could work on indexers or heavy forwarders.

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

jcoates_splunk — Sat, 17 Oct 2015 20:54:40 GMT

meta... converted a couple of the clarification request comments into answers to clear the un-answered flag.

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

bworrellZP — Mon, 19 Oct 2015 12:28:38 GMT

Jcoates,

When I did the original install, there was a note that IF you where pushing out the app via a deployment server, that it had to be on the search head. That said, even in this document, once it is installed all the work is done by the indexer, as I have it.

Also, prior to the update to 2.04, the health tab worked fine.

So I have installed the app on the search head now. Still NOT using the deployment app portion, as there does not appear to be a need based on the document. I only have one server doing database pulls. Data pulls in fine (one issue but unrelated to the app).

Now with that said, any thoughts as to why neither show the Health tab or where to look to find the errors?

Thanks

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

bworrellZP — Wed, 28 Oct 2015 14:00:47 GMT

So, issue is not resolved, comments did not offer any suggestions.

Looking elsewhere, found the log path. Seems that the py_health.log file hasn't had any updates in almost two months. Since I updated to 2.0.4.

Seems the health dashboard needs that log to update.

Anyone else have this issue?

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

bworrellZP — Tue, 29 Sep 2020 07:43:45 GMT

Answer is - Seems there is a Bug in the DB Connect client since 2.0.5. When the py_health.log hits 10,000 kb, the file locks, it does not rollover or purge as it should. That causes errors in the splunkd.log about mi_input.py being in use by another process.

Once I purged part of the file, the health data showed up.

Had nothing to do with the suggestion by jcoates (hence why I was able to get Health stats prior to 2.0.5.)

Just a bug in the DB Connect client. Will watch the log and clean it out until issue is fixed.

Re: Why is the Splunk DB Connect 2.0.5 Health dashboard not populating data?

HMTODD — Fri, 20 Oct 2017 17:18:15 GMT

Thanks you for this! Just ran into the same problem. Purged records from the py_health.log file and dashboard began to populate. Anyone know if Splunk intended to fix this bug?