topic Re: Why is Splunk DB Connect 3 days behind indexing ePO data? in All Apps and Add-ons

Why is Splunk DB Connect 3 days behind indexing ePO data?

bluemarvel — Fri, 17 Mar 2017 20:09:34 GMT

We are seeing logs in Splunk but there are lagging; meaning they are always 3 days behind. I.E we can only see logs from the 14th and prior. Is there a some time format or other configuration that needs to be changed?

Re: Why is Splunk DB Connect 3 days behind indexing ePO data?

gjanders — Fri, 17 Mar 2017 22:28:49 GMT

As per the documentation https://docs.splunk.com/Documentation/DBX/3.0.1/DeployDBX/Createandmanagedatabaseinputs is it possible your max rows to retrieve is too small?
There is also a query timeout of 30 seconds within the db_inputs.conf you could possibly be hitting https://docs.splunk.com/Documentation/DBX/3.0.1/DeployDBX/dbinputs

Re: Why is Splunk DB Connect 3 days behind indexing ePO data?

bluemarvel — Tue, 29 Sep 2020 14:46:55 GMT

thank you for the advice, much appreciated. Still having and issue after I changed the max_rows value to all, (max_row = all) which is the default from what i have read. >>{/splunk/etc/apps/dbx/local/inputs.conf}

Re: Why is Splunk DB Connect 3 days behind indexing ePO data?

gjanders — Tue, 29 Sep 2020 13:18:29 GMT

Do the DB Connect logs in /opt/splunk/var/log/splunk/splunk_app_db_connect_* give you any hints?
You can also find them in the internal index, they might tell you waht is happening...

Furthermore, are you sure the records are old ? i.e. do you have an issue parsing the timestamps/using the wrong time column or similar ?

Re: Why is Splunk DB Connect 3 days behind indexing ePO data?

bluemarvel — Tue, 29 Sep 2020 13:18:40 GMT

hello again, no there are no error messages in the dbx.log

======================================================================================
2017-03-20 11:45:34.916 dbx4196:INFO:TailDatabaseMonitor - Database monitor=[dbmon-tail://epo_b-edc/epo_b-edc_5] finished with status=true resultCount=10000 in duration=886266 ms
2017-03-20 11:45:34.916 dbx4196:INFO:ExecutionContext - Execution finished in duration=886266 ms

2017-03-20 11:45:34.916 monsch1:INFO:Scheduler - Execution of input=[dbmon-tail://epo_b-edc/epo_b-edc_5] finished in duration=886266 ms with resultCount=10000 success=true continueMonitoring=true

what is happening is that the logs are coming, but they are three days behind, so today is the 20th, we can see logs from up to the 17th only. Tomorrow the 21st we will see logs up to the 18th.

Re: Why is Splunk DB Connect 3 days behind indexing ePO data?

gjanders — Mon, 20 Mar 2017 23:20:33 GMT

The logs indicate:
resultCount=10000

Did you try increasing this number to see if you just need more data per-request? Do you have more than 10K of rows per day? Or alternatively you can set the query to run more often/more times per day?

Re: Why is Splunk DB Connect 3 days behind indexing ePO data?

bluemarvel — Tue, 21 Mar 2017 18:34:54 GMT

is this also in the local.con file because I am not seeing it.
the other config change that I made proved not have worked.

Re: Why is Splunk DB Connect 3 days behind indexing ePO data?

bluemarvel — Mon, 27 Mar 2017 21:38:52 GMT

So we decided to move on to DB Connect 2 for our EPO events, however I am seeing two different database drivers for this, whicn is the right one?

sqljdbc4.jar and postgresql-9.4.1212.jre6.jar